Hi all,

I've been trying, somewhat unsuccessfully, to utilize a SCM tool to manage my projects, and it seems inevitable that I wind up with conflicts and/or data loss every few months or so. I started out trying to use GIT about a year ago, ran into a bunch of issues, found Fossil about 2 months ago and switched over to that, and I'm still running into similar issues. That means the problem is with me and my understanding rather than the tools.

That being said, I was hoping I could put myself out there and give my view of what I'm trying to accomplish and the general steps I'm using to get there, and get some pointers on what I'm doing wrong.

First off, I'm working alone, and I'm not trying to do anything across a network. I don't have any remote repositories that I'm syncing with. All of my SCM usage is done locally to try to manage "production" code along side "development" code. I have one top-level "Projects" directory with all of my sub projects in directories within it.

So, the way I'm trying to think of this is my trunk is my production code and I create a branch whenever I want to make a code change. My thought is that I want to be able to easily switch back and forth between the branch and the trunk as needed until I have completed my code changes in the branch, at which point I merge the branch into the trunk, commit the changes, and close the branch.

This usually works well, but I've found that I run into merge problems if I've made changes in multiple branches and try to merge them back into the trunk.

For example, say I want to create a new project, so I create the new branch ("newproject"), start creating files and editing code, and then I realize I need to make a quick change to an existing project to fix a bug, so I commit the new project where it is, update to trunk and create a second branch ("bugfix"), edit that code, commit it to the bugfix branch, update back to trunk, merge it in and commit it to the trunk, and close the bugfix branch.

Now I go back to the newproject branch, finish the project, commit it to that branch, update to trunk, try to merge it, and I get a message about a merge conflict because the new project contains an old version of the file I fixed in the bugfix branch and already merged to the trunk.

In case the prose version is too convoluted to understand, this is what I'm doing, starting at the initial "everything's working" point of the trunk:

1. fossil branch new newproject trunk
2. [create and edit files for newproject]
3. fossil addremove
4. fossil commit -m "Pausing to work on a bug in another project"
5. fossil update trunk
6. fossil branch new bugfix trunk
7. [edit file for bugfix]
8. fossil commit -m "fixed bug"
9. fossil update trunk
10. fossil merge --integrate bugfix
11. fossil commit -m "bugfix merged to trunk"
12. fossil update newproject
13. [continue to create and edit the project]
14. fossil addremove
15. fossil commit -m "Completed newproject."
16. fossil update trunk
17. fossil merge --integrate newproject (This is where I get my merge conflict)

In my head, I'm thinking this should be a non-issue because I didn't actually make any changes to that file in the newproject branch. When I do a merge from newproject to trunk, I'm expecting that it would ignore that file because I didn't actually make changes to it in newproject.

Is this the wrong way to be thinking about it? Even though I've read branching/forking part of the Fossil docs multiple times, I still don't understand the conceptual difference between forking and branching, so I'm not sure if I'm doing something completely wrong. Am I missing something with this whole process? Maybe I'm just not understanding what a SCM is for or how to use it correctly, but I'm happy to for any pointers or direction anyone can give.

Thanks!

So you are expecting that once the bugfix branch is merged into trunk, your already existing branch newproject should be unaffected by what changes occurred in bugfix, and there should be no merge conflicts when finishing up newproject and merging to trunk?

Maybe try explaining the problem you want to solve is. Essentially: switch between branches without merge conflicts on unrelated files?

Maybe try explaining the problem you want to solve is. Essentially: switch between branches without merge conflicts on unrelated files?

My apologies. I guess my point got lost in the description. Yes, that's exactly what I'm trying to do. My thought is if the trunk has a modified file, but that same file is unmodified in a branch, when I merge the branch back to the trunk, the newest update (trunk version) would "win", in a sense.

That is how it works. You only get a merge conflict when both branches have modified the same file near the same point in the file.

If both branches have modified copies but the changes are far enough apart — ≥3 lines, I believe — then Fossil will be able to merge the two sets of changes automatically.

The command fossil merge trunk can be helpful on short-lived branches to limit the diffs between them to just the essentials. It's pointless at the start of the branch, but it can be helpful if both branches get checkins to the same files.

On an unusually long-lived but temporary branch, I will often merge trunk just before attempting to merge the branch down into trunk and close it (--integrate) as a check that the branch will merge cleanly.

Ok, that makes sense. Then I must have made an update to the file while checked into the branch without realizing it. I realized that while responding to wyoung. But I was also missing the fossil merge trunk while in the branch, so that will be helpful going forward.

Fossil, in my opinion is a better choice for personal and small team projects. So, I think you made a good choice when switching to Fossil.[1]

I think the reason Fossil (or git) is complaining of a merge conflict is that you are merging from the branch to the trunk.

When I am working on a branch and I know the trunk has been updated since I started my branch, I first merge from trunk to the branch. then I rebuild and test the branch, make any needed changes, then make the final commit to the branch and merge it to trunk. This way, the only changes being merged in are ones that originated in the branch.

In the branch:

    fossil merge trunk
    # resolve any conflicts
    make
    # run tests
    # fix new problems
    # make/test/fix - repeat as needed
    commit -m "Merge changes from trunk"

Then in trunk:

    fossil merge branch
    # make/test/fix as needed
    fossil commit -m "New feature from branch"

This approach especially makes sense when the trunk is being treated as "production ready".

Also, there is a way to simplify your workflow.

One of the things I like about Fossil is that you can easily have multiple working copies with a single (local) repository.[2]

Anyway, I find this helpful because I can have a working copy open for each branch I am currently working on.

If I find a bug (or some other need come ups), I can create a new branch then open the new branch in a new working copy. This way, (a) I don't have to worry about files not yet added to the repository, and (b) I'm not constantly switching a working copy between branches.

For example:

    cd ~/Projects/MyProject/trunk
    fossil branch new bugfix
    cd ..
    mkdir bugfix
    cd bugfix
    fossil open ../repo.fossil bugfix

(I have the above scripted. I just cd to the trunk and do "newbranch bugfix".)

Once I am finished with a given branch, I can delete the associated working copy.

[1] Fossil can be and is used for very large projects. As I recall, at least one of the BSD Unix projects uses Fossil. git was designed for global scale teams, with thousands of developers, by people running a global scale project.

[2] Although git can do this, it was designed to for every working copy to have its own repository, so this capability had to be grafted in.

In the branch:

fossil merge trunk

# resolve any conflicts

This is the step I was missing! I didn't realize that the trunk needed to be merged into the bugfix/feature branch before changing over to the trunk and merging in the branch.

If I find a bug (or some other need come ups), I can create a new branch then open the new branch in a new working copy. This way, (a) I don't have to worry about files not yet added to the repository, and (b) I'm not constantly switching a working copy between branches.

For example:

cd ~/Projects/MyProject/trunk

fossil branch new bugfix

cd ..

mkdir bugfix

cd bugfix

fossil open ../repo.fossil bugfix

I cannot, for the life of me, wrap my head around what opening the repo for the second time is doing or supposed to be doing. I think someone else mentioned this, too, so it's obviously "a thing", but I'm just going to have to try it out on some test code and figure out what's going on with it.

hi, before you merge 'newproject' to trunk you must merge trunk into newproject fix conflicts and then merge newproject into trunk, this allow conflict not affect your trunk. Don't worry nobody has the last word, just make your own way.

[branch newproject]
16. fossil merge trunk
17. [fix possible conflict]
18. fossil update trunk
19. fossil merge newproject

for pausing to fix you can use 'fossil stash' also fossil allow open multiple times same repository 'fossil open'.

$ mkdir app-newproject
$ mkdir app-production
$ cd app-newproject
$ fossil open ../mirepo.fsl newproject
$ cd ../app-production
$ fossil open ../mirepo.fsl production

thanks

[branch newproject]

1. fossil merge trunk

2. [fix possible conflict]

3. fossil update trunk

4. fossil merge newproject

Ah! Yes, I think the initial "fossil merge trunk" was what I am missing. It's a simple thing, but it's something I very well may have missed in the docs when trying to figure out branches. Thank you!

Also, regarding fossil stash, I've tried that, but it's too easy for me to be careless and apply the stash to the wrong branch. I ran into the same thing with GIT, more than once popping a stash into the wrong place and having a bad time trying to get it cleaned up. I would prefer to use the stash option instead of a commit as it would be cleaner in the timeline at the end. But since the timeline is just for me and not shared with anyone else, I'm ok with the ugly non-functional commits at the moment.

Part of me wishes that a stash would only apply to whatever branch I save it from, or at least there could be some sort of flag to force it to be applied to a different branch, but I get that I'm the newb and things like this are long established, so I just need to get my head wrapped around it better.

I don't have any remote repositories that I'm syncing with.

Even in a single-user system, I'd encourage you to at least have the repository server on a different machine on the LAN than your development system, simply as part of your backup and recovery strategy. A Fossil checkin should feel good at a deep psychological level for the same reason that File → Save does.

You should then have a remote backup of some kind. If security is a concern, consider something like this pseudo-shell command:

  $ fossil sql .dump | xz -9c | gpg -c > ~/Dropbox/backupfile

If Dropbox (in this example) ever exposes your data to someone you don't want to see it, they see only a blob of pseudorandom noise, because they lack the GPG key.

In this way, the chance of actual data loss is very near zero.

I have one top-level "Projects" directory with all of my sub projects in directories within it.

I'm not sure if you mean "project" the same way as in common IDEs, to mean a long-term self-contained development effort. From later parts of your post, I wonder if you consider "project" and "feature" to be near-synonyms.

Lacking any clear guidance, I'm going to go with the first definition by default in my answers below, being the more broadly accepted.

You'll have to adjust the interpretation of my answers if I've guessed wrong.

my trunk is my production code and I create a branch whenever I want to make a code change

I'm very much opposed to that, especially for a single-developer project.

Trunk is special. There is only one. If there is only one developer, why then would you not use trunk as the main line of active development, and shunt code off onto branches when it's more nearly finished?

You can expand the argument to multi-user teams, since you probably still have one main line of development, which naturally maps to Fossil trunk.

The primary rule I have for trunk is that it always build, at least on the development system where the checkin was made. If you have a test suite (and if not, why not?) checkins to trunk should also always pass the tests at the time of the checkin.

Branches have two primary purposes:

1. Short-lived branches are for temporary bits of work where you can't make a change in a single well-focused Fossil checkin that conforms to the above rule: checkins never break trunk. You make your series of checkins on this branch over the course of hours to weeks at most, then merge the result back into trunk. It is important that you merge back into trunk ASAP, else you risk creating a tangent that goes off into space and never comes back.

   This may be what you mean by "project," and in that case, I think our development practices aren't all that far apart after all.

2. Long-lived branches are for taking a snapshot of trunk with the intention of stabilizing it over arbitrarily long periods of time. Primary among these are your stable major version branches: v1, v2, v3, etc. Once trunk forks off to a stable branch, destabilizing work happens on trunk, and only "safe" work happens on the stable branches.

   Whether you choose to check such work in on the branch and merge down to trunk or do it on trunk and cherrypick it up onto the branch(es) that needs it isn't very important; please yourself. Personally, I do it both ways based on local conditions at the time I make the decision.

   A common situation with long-lived branches is that a user running v2 comes to you and says they need a fix for a bug they've discovered, so you develop a fix on trunk, test it there, check it in, then cherrypick it onto the v2 and v3 branches, re-testing on each, then check it in separately on each. Now you can cut a new v2.x.y version from the v2 branch and send it to the user that reported the bug. (Trunk, in this case, is probably the in-development work that will lead to a v4 brach.)

   Some projects don't need multiple long-term release branches. For that, I just call the branch "release". Work happens on trunk until I'm happy enough with it that it's worth calling it a release, then I merge everything into the release branch and resume work on the trunk.

say I want to create a new project, so I create the new branch

If you're using my first definition of "project" above, why are you not creating a new Fossil repository instead?

There are possible good answers, like "Because the new project shares a substantial amount of code with the prior project(s), which is already factored out into reusable libraries, which are also checked into that same repository."

If "project" and "feature" are near-synonyms to you, then please disregard the question.

fossil branch new newproject trunk

I'm not sure I've ever used the command fossil branch. It's always fossil ci --branch for me, meaning that the branch is only created upon the first checkin to that new branch.

fossil commit -m "Pausing to work on a bug in another project" ; fossil update trunk

That's Git-style thinking there. As a rule, you should never switch branches within a working directory with Fossil. The primary exception is the briefest sort of short-lived branching as described above.

Branches with lifetimes longer than a day or so should have a separate checkout directory. You can then switch branches with cd, not fossil up:

  $ mkdir -p ~/Projects/repo/trunk
  $ cd ~/Projects/repo/trunk
  $ fossil open ~/museum/repo.fossil trunk
  $ mkdir ../v1
  $ cd ../v1
  $ fossil open ~/museum/repo.fossil v1

Now you have two checkout directories under ~/Projects/repo, one for the Fossil trunk and one for the v1 stable release branch. Add as many as you need to keep up with active branches.

I will often also create a scratch checkout directory which does get treated as you show, switched around with fossil up commands. I use it when I need a temporary checkout of one branch that isn't given its own long-lived checkout directory. Often this is one of the short-lived branches I've talked about. This is a common workflow for me:

  $ cd ~/Projects/repo/trunk
  ...develop first cut at a new feature...
  $ fossil ci --branch my-new-feature -m "First cut at new feature"
  $ fossil up trunk          # "undo" checkin in this working directory
  $ cd ../scratch
  $ fossil up my-new-feature
  ...continue work on new feature...
  ...hours to weeks of work; and then...
  $ cd ../trunk
  $ fossil merge --integrate my-new-feature
  ...build and test...
  $ fossil ci -m "Integrated my-new-feature branch"

I didn't actually make any changes to that file in the newproject branch.

You must have. Fossil isn't complaining without cause.

A merge conflict means that at least one file has changed in both the source and target branches of a merge, and that the changes in each are close enough together that there isn't sufficient context for the Fossil merge algorithm to stitch the two changes together.

If this is happening in a text file, search for lines containing <<<< and >>>> markers. You'll see changes from both versions, showing how each branch sees that area of the file.

If you remain firm in your belief that there is no substantial difference in that file between the two versions, copy the file from a checkout of the merge's source branch, then run a fossil diff on it to be sure that the changes — for there must be changes for a conflict to occur — are insubstantial, however you define that for this repo:

  $ pwd
  ..../repo/newproject
  $ cd ../trunk
  $ fossil merge --integrate newproject       # merge conflict in foo/bar/baz.c
  $ cp ../newproject/foo/bar/baz.c foo/bar
  $ fossil diff foo/bar/baz.c | less

If you don't have a ../newproject checkout directory, because it's a short-lived branch, use your scratch directory. Insert this before the second-to-last command above:

  $ cd ../scratch
  $ fossil up newproject
  $ fossil revert         # needed only if this creates *another* merge conflict
  $ cd ../trunk

I apologize if I've given the impression that I'm any sort of "real" developer. (Good grief that sounds terribly sarcastic, but I do mean it sincerely. :-) ) I mainly deal in SQLite scripts and Python scripts, and I'm just self taught and scraping by. As such, sometimes things break when I move them to a new directory, which is why I try to avoid it and utilize a SCM tool to develop "in place". That's a whole different issue for me to address, I know, but I'm working on it slowly.

You should then have a remote backup of some kind.

I agree, I should. I do, in a way, in that I'm on a Mac and have a TimeMachine backup going to a remote server. I know it's not what you're talking about, but at least it's something. I am planning on addressing it more directly if I can get my usage of fossil down to something reliable. (i.e., if I can stop screwing it up.)

I have one top-level "Projects" directory with all of my sub projects in directories within it.

I'm not sure if you mean "project" the same way as in common IDEs, to mean a long-term self-contained development effort. From later parts of your post, I wonder if you consider "project" and "feature" to be near-synonyms.

Lacking any clear guidance, I'm going to go with the first definition by default in my answers below, being the more broadly accepted.

My "projects" tend to only be a file or two and completely unrelated to one another, living in separate directories. I switch between them frequently enough that to try to maintain a separate repository for each one would add to the complexity that I'm already trying to avoid. They're Python and SQLite scripts that don't require building or compiling. They just get pointed to the interpreter executable to run, which is why I treat my trunk as "production" and branches as "development". I need to have a working copy that I can switch back to in order to run it even if I'm in the middle of a bugfix or feature enhancement.

my trunk is my production code and I create a branch whenever I want to make a code change

I'm very much opposed to that, especially for a single-developer project.

Trunk is special. There is only one. If there is only one developer, why then would you not use trunk as the main line of active development, and shunt code off onto branches when it's more nearly finished?

That's an interesting way of thinking about it. Maybe it's a leftover from trying GIT first, but my thinking was that trunk is special because it's where the "final" code lives. For the work I'm doing, it would mean that whenever I want to run a particular script, I would need to change to that branch in order to get to something that's in a stable state. I'm not sure if that would be more painful than trying to do it the way I'm currently doing it, but it's definitely something to consider.

If you have a test suite (and if not, why not?)

I'm assuming you are meaning something more than "try running it and see if it crashes or doesn't produce the desired results." I have no idea what something more than that would be. Each script is so different, I don't have a single way to automatically test all of them. That's probably beyond what this post was about, anyway, since it's not fossil related, but it just goes back to my point of "self taught and scraping by".

Short-lived branches are for temporary bits of work where you can't make a change in a single well-focused Fossil checkin that conforms to the above rule: checkins never break trunk. You make your series of checkins on this branch over the course of hours to weeks at most, then merge the result back into trunk. It is important that you merge back into trunk ASAP, else you risk creating a tangent that goes off into space and never comes back.

This may be what you mean by "project," and in that case, I think our development practices aren't all that far apart after all.

Yes! This is exactly what I'm trying to do, because it's rare that something I'm working on takes more than a week or two. This goes back to what I was saying about feeling like the trunk is where the stable code lives.

I'm not sure I've ever used the command fossil branch. It's always fossil ci --branch for me, meaning that the branch is only created upon the first checkin to that new branch.

I do it that way because I don't want to accidentally commit directly to the trunk, especially if I'm near the end of the day and need to put things on hold to get ready for tomorrow. I always need my projects in a spot where they are able to run first thing in the morning. By creating the new branch up front, I mentally put myself in a "safe space" where the production code is still available when I need it.

$ mkdir -p ~/Projects/repo/trunk

$ cd ~/Projects/repo/trunk

$ fossil open ~/museum/repo.fossil trunk

$ mkdir ../v1

$ cd ../v1

$ fossil open ~/museum/repo.fossil v1```

Someone else mentioned opening the repository multiple times in order to accomplish something like this, but I can't wrap my head around what that's actually doing, or even what it's trying to do. Being new-ish to all of this, it's one of those things that I'll just have to test and figure out another time.

I didn't actually make any changes to that file in the newproject branch.

You must have. Fossil isn't complaining without cause.

This is probably correct. I just learned about the "fossil changes" command the other day, so I'm trying to get into the mental habit of running that before I do a commit, which will be helpful for catching mistakes.

When going through the documentation, it's been hard for me to pick out what the workflow should look like. It feels like there are two facets to the docs. The first is the "extremely simple" case where it's going over how to create a new repository and how to commit. Then it jumps to stuff that it feels like you need to be familiar with from using SCMs previously. There's not much of an in-between. Like I said in my original post, evidently there's a huge distinction between branching and forking, but I still have no idea what it is, even after having gone through that doc numerous times.

Thanks for all your help and input! You've given me lots to think about, and it's very valuable information. Regarding the point of my original post, I think bit4bit got me pointed in the right direction with needing to do a merge from trunk to the branch first to check for file conflicts, and then merging back to the trunk with the new and final code.

When going through the documentation, it's been hard for me to pick out what the workflow should look like. It feels like there are two facets to the docs. The first is the "extremely simple" case where it's going over how to create a new repository and how to commit. Then it jumps to stuff that it feels like you need to be familiar with from using SCMs previously. There's not much of an in-between. Like I said in my original post, evidently there's a huge distinction between branching and forking, but I still have no idea what it is, even after having gone through that doc numerous times.

Which brings back to me a feeling I had a couple of times before: The new and prospective Fossil users (or programmers) would really benefit from usage guides/testimonials other than the quick start and alike. My impression is that most documents are technical - which is really nice - and we need more of them for the general user. We should not forget that many advantages that make Fossil so wonderful (e.g. one executable + one database) do appeal to the young or unexperienced programmer. And we do want that all programmers use version control, don't we?

Maybe the more experimented Fossilers around here can put together such documents - how they structure a versioned project, when and where they branch (from and back, private or not), stash; for what they use specific branches (trunk, short/long lived) or parallel working directories etc.

(The second round can touch on cherry-pick/backout, grepping, bisecting, or anything which has proved regularly useful.)

There's a lot of experience here that can be shared and many mistakes that a lot of new users can spare. (Not that it's not true everybody benefits from their own mistakes, but there's no shortage of that anyway.)