bugQuilt - Bugs: bug #53444, Please allow more flexible...

 
 

bug #53444: Please allow more flexible installation of patches (dependencies instead of strict patch order)

Submitter:  Filipus Klutiero <chealer>
Submitted:  Fri 23 Mar 2018 05:24:21 PM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Need Info
Privacy:  Public Assigned to:  khali
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 16 May 2018 09:04:36 PM UTC, comment #12: 

This is a follow-up to comment #4.

While guards can work with many pseudo-series files, it expects each line to contain nothing but a path. So, if for example a line reads...
Backport_FP#5701.diff -p0
... the generated series files would lose the "-p0" switch. I do not see any existing option to avoid that. So guards is not compatible with our patchset.

Filipus Klutiero <chealer>
Thu 12 Apr 2018 06:24:39 PM UTC, comment #11: 

This is a reply to comment #8.

> Quilt is not aware of dependencies beyond the order of the patches in the series file. "quilt graph" can be used to try and guess the dependencies, but its heuristics are very basic and it is not meant to be complete (and it can't be, really.) The user is supposed to know the dependencies, and to sequence the patches accordingly in the series file.


This is probably a different way to describe the same situation, but I would say that in the current situation, the "user" which runs quilt does not necessarily have to know dependencies. It is the developer which creates series files which has to know. I would say that Quilt is aware of dependencies, in a sense, but it is indeed not aware of the exact (minimal) dependencies of each patch.
 

> Your last example is utterly confusing. It looks pretty theoretical to me, almost academic, considering that "diff" and "patch" operate on a per-line basis, so consecutive changes on the same line must have a well-defined order by definition. Whether these changes depend on each other is irrelevant then, due to this technical limitation.


I intentionally used a trivial example which is indeed unlikely, but there are numerous similar cases. For example, starting from:
<span>Choose files</span>
... patch 1 may translate with:
<span>{tr}Choose files{/tr}</span>
... and patch 2 may bring to:
<div>{tr}Choose files{/tr}</div>

The corresponding patches could "conflict", as patch is not smart enough to apply 2 patches which would both be based on the original line.

So, either:
1. As I suggested, if patch #2 is based on the file modified by patch #1, it is declared as "artificially depending" on patch #1.
2. Quilt automatically figures out artificial dependencies between patches, which may turn out difficult.
3. Quilt tries applications and may report to users that the operation they requested failed.
 

> You are focusing on the dependency model, and this is indeed one thing that would have to be extended in quilt to support your scenario. However this is not the only problem, and probably not the most difficult one.
> For me, the real problem is that quilt was fundamentally designed around the idea that the set of applied patches always matches the beginning of the series file. As a matter of fact, we even have a check in place to stop any quilt operation except "pop" whenever this assumption is no longer true.
> This design decision has two implications:
> 1* We know that all patches below the top patch must be applied, and all patches above the top patch are not applied, always. This allows for code optimization in various places. To be able to apply the patches in arbitrary order, we would have to make the code slower and/or more complex.
> 2* The semantics of the quilt commands are clear. If suddenly it is possibly that the series file and the list of applied patches are unrelated, the meaning of several commands will become ambiguous. For example, would "quilt previous" return the patch before the top-most patch in the series file, or the patch that was applied right before the topmost patch? Would "quilt next" return the patch after the topmost patch in the series file, or the first unapplied patch in the series file? With the current model, it doesn't matter because the answer is the same. With your proposed model, the answer is no longer the same.


I agree that in addition to a dependency model, a new format to specify dependencies needs to be determined.

I basically agree with point 2, although the term "ambiguous" may be strong, and I suppose quilt next; would have to consider the suggested order, it cannot guess the actual installation order. For quilt previous; the manual can of course document what it does. Or an argument allowing to choose between the 2 possibilities can be added.
 

> So in order to implement your request, we would not only have to add support for explicit dependencies between patches. We would firstly have to redesign the quilt command set, and rewrite the underlying code. I can't see this happening, sorry. Too much work and too risky, only to solve what is essentially an organizational issue in your company really.


I would not say that this request is essentially to help with an organizational issue in my customer's organization.

First, I think there are legitimate reasons why server administrators and developers are kept separate. For example, if we developers create a bug which causes problems, we can't rush to deploy a fix hoping to cover our error without other teams noticing. I am sure that many organizations work this way.

Second, there are simpler reasons why solving this problem would help us. For example, if I notice a new problem and try to identify which patch it comes from, I am likely going to want to remove a single patch at some point. If there are no patches which really depend on it, it would help to be able to remove it without removing other patches just because these come later in series.

Thinking about other projects, if Bob is a Debian user which develops Mozilla products, perhaps Bob will want to run a Firefox as close to Mozilla's as possible, and only apply the really necessary patches. But if some patches come late in series, Bob would have to modify series to unapply them, which may not be something Bob is confident enough to do.

I am confident that solving this would be beneficial to some degree for a large portion of users. But I absolutely agree that it needs important changes, which would take even more important work in a shell scripting context. I would certainly advise anyone planning to write a solution to first describe it and seek pre-approval from Quilt maintainers.

Filipus Klutiero <chealer>
Thu 12 Apr 2018 03:55:55 PM UTC, comment #10: 

Thank you for your replies and for your commit Jean, here is a reply to comment #9.

There is nothing in the guards man page which I find unclear, but traditionally manual page are minimalist. Extensive documentation is often available elsewhere.

The documentation which I would like to find somewhere - not necessarily in the manual page - is examples of uses. For example:
1. Without guards, organization foo has problem p.
2. Organization foo goes from a set of series files x to a set of guarded series files y.
3. Organization foo calls guard in ways z to solve problem p.

Adding this may be too much for the manpage, but my personal preference about manpage is to have at least one example call to the documented command, so I think it would be good to add a little something.

Is it possible that your example in comment #4 has its 2 commands inverted?

Filipus Klutiero <chealer>
Thu 12 Apr 2018 08:02:48 AM UTC, comment #9: 

It is true that "guards" was not advertised anywhere. As a first step, I have mentioned it in the quilt man page:

http://git.savannah.nongnu.org/cgit/quilt.git/commit/?id=cc93a42248ce58f8cf320dc88453a723b79da301

I am not aware of any other documentation for "guards" than its manual page, but I'm also not sure why there should be something else. It's the very purpose of the manual page to explain how to use the tool, so if anything is not clear, the manual page should be improved.

Is there anything in particular in the guards man page which you find not clear? Or anything missing?

Jean Delvare <khali>
Group administrator
Thu 12 Apr 2018 07:56:47 AM UTC, comment #8: 

Quilt is not aware of dependencies beyond the order of the patches in the series file. "quilt graph" can be used to try and guess the dependencies, but its heuristics are very basic and it is not meant to be complete (and it can't be, really.) The user is supposed to know the dependencies, and to sequence the patches accordingly in the series file.

Your last example is utterly confusing. It looks pretty theoretical to me, almost academic, considering that "diff" and "patch" operate on a per-line basis, so consecutive changes on the same line must have a well-defined order by definition. Whether these changes depend on each other is irrelevant then, due to this technical limitation.

You are focusing on the dependency model, and this is indeed one thing that would have to be extended in quilt to support your scenario. However this is not the only problem, and probably not the most difficult one.

For me, the real problem is that quilt was fundamentally designed around the idea that the set of applied patches always matches the beginning of the series file. As a matter of fact, we even have a check in place to stop any quilt operation except "pop" whenever this assumption is no longer true.

This design decision has two implications:

1* We know that all patches below the top patch must be applied, and all patches above the top patch are not applied, always. This allows for code optimization in various places. To be able to apply the patches in arbitrary order, we would have to make the code slower and/or more complex.

2* The semantics of the quilt commands are clear. If suddenly it is possibly that the series file and the list of applied patches are unrelated, the meaning of several commands will become ambiguous. For example, would "quilt previous" return the patch before the top-most patch in the series file, or the patch that was applied right before the topmost patch? Would "quilt next" return the patch after the topmost patch in the series file, or the first unapplied patch in the series file? With the current model, it doesn't matter because the answer is the same. With your proposed model, the answer is no longer the same.

So in order to implement your request, we would not only have to add support for explicit dependencies between patches. We would firstly have to redesign the quilt command set, and rewrite the underlying code. I can't see this happening, sorry. Too much work and too risky, only to solve what is essentially an organizational issue in your company really.

Jean Delvare <khali>
Group administrator
Mon 09 Apr 2018 10:05:01 PM UTC, comment #7: 

After more thought, I'd suggest considering 2 types of "dependencies" between patches first.diff and second.diff ...

A first type would be a real dependency. If the change performed by first.diff is missing, applying second.diff may work, but the resulting application will be broken.

The second type would be an "artificial" dependency. If the change performed by first.diff is missing, quilt/patch will refuse to apply second.diff, because both modify the same lines. But if second.diff was rewritten so it doesn't contain the initial lines as modified by first.diff, it could be applied without breaking the resulting application.

An example of the second type would be if the program writes the message "Helo Word". first.diff would add the missing "l", and second.diff would add the missing "l". second.diff could be considered "dependent" on first.diff, but in reality, any fix can be made first.

Filipus Klutiero <chealer>
Mon 26 Mar 2018 08:34:13 PM UTC, comment #6: 

Thank you Jean for comment #4,
It is true that dependency-based ordering could apply changes at the wrong place of a file, but that is already possible and there are ways to mitigate the risk.

In terms of actual description of our needs, as discussed in my previous comment (#5), I need to provide colleagues with the ability to apply patches in more than 1 order, so there is some flexibility in deployment, without having to develop multiple series files nor allowing others to modify series files.

It is true that dependencies does not tell which patches need to be applied. Our administrators may run $ quilt push foo.diff; only, but it would still be nice to keep the ability to run $ quilt push -a;.

The "guards" tool apparently does not deal with dependencies, but thank you for mentioning it, I was not aware of its existence. If there is more documentation than its manual page, it would be appreciated.

Filipus Klutiero <chealer>
Mon 26 Mar 2018 08:19:29 PM UTC, comment #5: 

Thank you Martin,
The complexity of our context is bureaucratic. I am part of the software team, but I have no control over the business team, which determines whether a change can be deployed or not. We also have an infrastructure team which would be the one applying patches to the production instance. The infrastructure team knows nothing about our software, nor even our technologies. We don't mind the infrastructure team running quilt push/pop, but we don't want them to determine (effectively) patch file dependencies by crafting series files themselves. What we want is to provide a flexible set of files. Then, if the business team tells the infrastructure team to apply some patch, the infrastructure team can do so without modifying files nor requiring the software team to intervene.

Filipus Klutiero <chealer>
Sat 24 Mar 2018 08:51:45 AM UTC, comment #4: 

A tool which would apply patches in apparently random order based on resolution of explicit dependencies sounds frightening to me. Applying patches is tricky. You could easily end up applying changes at the wrong place of a file without noticing - even if you got the dependencies right. If you have many patches, maintaining the explicit dependencies could be challenging as well.

I also don't see an actual description of your needs in this bug report. You focus on dependencies and ordering. Having explicit dependencies attached to patch files is one thing, but it does not tell you which patches need to be applied and which patches should be skipped, which, as I understand it, is you actual problem to solve.

If I had to solve such a problem, I think I would simply use the "guards" tool which comes with quilt. You can use guards to generate an effective series file based on a template series file where each line can be guarded by an optional symbol. You would simply add patches with a "+needs_approval" guard, and remove the guard once the patch gets approved by the user department. That way you can generate two series files from the template. One with all the patches, for the developers:
$ guards +needs_approval < series.guarded > series
And one with only the approved patches, for production use:
$ guards < series.guarded > series

I suggest you give a try to "guards" and see if it works for you. It is generic and configurable, I believe you can get it do to what you need.

Jean Delvare <khali>
Group administrator
Fri 23 Mar 2018 09:11:41 PM UTC, comment #3: 

Hello,

your proposal would be rather complex to develop in a generic way, regardless of the programming language. I fail to see how your usecase, that is really simple compared to other settings in which quilt is successfully used (kernel both upstream and in all distribution, any software in debian and derivatives, etc), is really mandating such a difficulty.

Given that most of your patches do not depend on each other, why don't you just throw their names in the serie file?

If you need something more complex, I'm sure you can generate the serie file on your side using a specific formatting of the header. Simply add a Depends-on: field in the DEP3 [1], develop a simple tool in the programming language of your choice that parses these fields in all patches to generate the serie file before calling quilt, and you're set.

This is the good thing about the current implementation of quilt. Its format is so simple that it's really easy to extend its tooling on your side. For example in Debian, quilt is used by maintainer to modify the patches, but at build time, they get applied by dpkg directly without depending on quilt itself. Plus, the DEP3 standard allows a lot of additionnal tooling that quilt has no idea about.

Thanks for your feedback anyway -- my opinion is in no way definitive here.

Bye, Mt.

[1] http://dep.debian.net/deps/dep3/

Martin Quinson <mquinson>
Group administrator
Fri 23 Mar 2018 05:36:59 PM UTC, comment #2: 
Filipus Klutiero <chealer>
Fri 23 Mar 2018 05:33:21 PM UTC, comment #1: 

I realize that solving this with software based on the shell command language would be challenging. If someone knows of similar software which may be more suitable for our use case, I would appreciate to know.

Filipus Klutiero <chealer>
Fri 23 Mar 2018 05:24:21 PM UTC, original submission:  

Context:
We are using quilt to manage local changes to a PHP web application. The vanilla engine has thousands of files. Our patches generally modify 1 to 4 of these files. Each patch developed by my team must be approved by the user department before being deployed. If our team creates patches #1, then #2 and then #3, the users may still approve patches #3, then #1 and only then #2. Depending on the time it takes them to test and their availability, the order of a series file would often not match the order in which we have to deploy the patches. Most of our patches do not depend on other patches.
---------------------------------------------------------------------------------------------

The biggest problem with using Quilt to manage changes in a context like ours would be that deployment of patches has to be done strictly in the order of patches in the series file. Quilt imposes a complete order and considered each patch as depending on the previous patch.

This can be worked around by documenting actual dependencies and shuffling patches in series when deploying, but letting the administrator do this is error-prone.

The safe solution would be to allow patches to declare dependencies on other patches, and to consider that if there are x patches, none of them specifying any dependency, then any of these x patches can be installed first. Patches would be considered like packages in operating systems such as Debian.

This does not prevent ordering patches, but if there is an order, it would only be suggested (a default).

So, technically, this is asking to add a Quilt "mode" for explicit dependencies, alongside the current mode based on implicit dependencies.

Filipus Klutiero <chealer>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by khali (Posted a comment)
  • -email is unavailable- added by mquinson (Posted a comment)
  • -email is unavailable- added by chealer (Submitted the item)
  • -email is unavailable- added by chealer
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-03-24 khali StatusNone Need Info
        Assigned toNone khali
    2018-03-23 chealer Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code