JavaScript Lockfiles

TL;DR: They're good! I'd love to use them unless there are compelling reasons not to.

Background

I've noticed that the vast majority of the SSBC JavaScript modules don't have lockfiles, and when PRs come along that add lockfiles I've seen people say "looks great, but can you remove the lockfile?". I've searched around and haven't really found a compelling reason not to use lockfiles, but the npm docs describes some pretty useful features of package-lock.json for libraries:

• Describe a single representation of a dependency tree such that teammates, deployments, and continuous integration are guaranteed to install exactly the same dependencies.
• Provide a facility for users to “time-travel” to previous states of node_modules without having to commit the directory itself.
• To facilitate greater visibility of tree changes through readable source control diffs.
• And optimize the installation process by allowing npm to skip repeated metadata resolutions for previously-installed packages.

What about applications who always want lockfiles respected with npm install? Run npm shrinkwrap and you get the above, plus the npm-shrinkwrap.json file gets published to npm. Are there any compelling reasons to avoid lockfiles? If not, I'd like to add lockfiles to our repos.

Libraries

Most of our modules are libraries, which work great with package-lock.json. My plan is to iterate through all repos configured as package-lock=false, remove that config, and add a lockfile.

Applications

When deploying npm modules with a CLI/TUI/GUI, I'd propose that we use npm-shrinkwrap.json so that dependencies remain locked. Currently, I think that's just Scuttlebot (soon to be ssb-server), which would alleviate the need to maintain scuttlebot-release.

I've found package-lock.json to be generally frustrating. Like, I can't tell if npm has installed from the package-lock or not. Also, package-lock.json only applies to npm install when you've checked it out, not npm install <module_name> but shrinkwrap does do that. So, that is why I have shrinkwrap in scuttlebot-release you can do npm install scuttlebot-release@version and actually get a known working version. (I'd recommend that anyone embedding scuttlebot in their application uses scuttlebot-release as is suggested in scuttlebot readme.

The other day, because of the unclarity of how npm handled package-lock I actually wrote my own install-from-shrinkwrap script, to install patchfoo. npm-install-shrinkwrap

Because of other problems, like noisy diffs, I'd rather just put shrinkwraps on the couple of modules that you might need to use independently, as -release or something. package-lock makes more sense for applications, though.

I've found package-lock.json to be generally frustrating. Like, I can't tell if npm has installed from the package-lock or not.

That sounds super frustrating, and I agree that the documentation isn't clear enough about how npm interacts with the lockfile. Is there a specific command that was behaving unexpectedly for you? My understanding is that npm only installs from the lockfile exactly when using npm ci or npm install module-with-shrinkwrap, but the rest of the time it follows this behavior:

1. The module tree described by the package lock is reproduced. This means reproducing the structure described in the file, using the specific files referenced in “resolved” if available, falling back to normal package resolution using “version” if one isn’t.
2. The tree is walked and any missing dependencies are installed in the usual fashion.

-- npm-package-locks (An explanation of npm lockfiles)

@cblgh

anyone have tips on improving that workflow?

I think npm update --depth 9999 multifeed should do the trick, does that work for you? If you drop "multifeed" from the end of the command it will be less specific and should update everything while respecting semver and avoiding duplication when possible. I was also using that workaround until recently, but I think it was only really required between npm 5.0 and 5.1 before this PR was merged.

Agree lockfile use by npm is confusing.
My opinion is that it's invaluable to have tool which we can at least start to reason about what might be different between your setup and mine.

I think my preference would be to make lockfiles default on, clarify as a community how we generally run install commands (so we're coding on similar looking things).
I don't know anyone using the scuttlebot-release ... it seems kinda odd given we have git tags as a tool (as long as they're pushed to github, which often they aren't). @dominic if you wanted to not install with a lockfile can you just delete it before npm installing? I guess I just don't understand why the master branch is not the release branch

@mix

I think my preference would be to make lockfiles default on, clarify as a community how we generally run install commands (so we're coding on similar looking things).

I'd also appreciate this. Looking through my history, I think this is all I use for installation:

• npm ci to get a known-working dependency tree
• npm install <module>[@"<version>"] to install/upgrade a module
• npm uninstall <module> to uninstall a module
• npm install whenever package.json changes
• npm --depth 9999 update <pkg> to get an update for a specific package
• npm audit fix whenever npm install complains about vulnerabilities
• npm update whenever I want to update everything (note: this changes package.json)

Are there any I'm missing? I think most of these are straight out of the npm docs, and I [personally] haven't had to do anything like rm -rf node_modules package-lock.json since npm 5.0 (currently on 6.4.1).

Are there any I'm missing?

Is

• npm dedupe

worth running?

@kas

Good recommendation!

My understanding is that npm does some deduplication by default, but that you can trigger a duplicate by installing dependencies in a specific order over time. For example, in January you could install something that depends on foo@1.x.x which resolves to 1.1.1, and then in February you could install something that depends on foo@1.2.x which resolves to 1.2.2, you could get a duplicate because npm tries to avoid a resource-intensive dedupe unless absolutely necessary.

• If you need the exact same version of two modules: if you need to assert that require('foo') in one module deep equals require('foo') in another module, you may need to manually dedupe.
• If you want to save space: this likely isn't an issue for most duplicate modules, but if somehow Electron gets duplicated or something then it would be silly not to dedupe.

I don't think I've ever been in any of the above scenarios, but it's definitely a useful tool because it saves others from having to dedupe (since their npm install will be driven by the lockfile).

Found a config to ignore lockfiles system-wide:

package-lock

• Default: true
• Type: Boolean

If set to false, then ignore package-lock.json files when installing. This will also prevent writing package-lock.json if save is true.

When package package-locks are disabled, automatic pruning of extraneous modules will also be disabled. To remove extraneous modules with package-locks disabled use npm prune.

This option is an alias for --shrinkwrap.

npm config set package-lock false

cc: @dominic is this something you'd fine helpful?
I think diffs can be ignored too:

# one-off command
git diff master -- ^package-lock.json

# global config
echo "package-lock.json binary" >> /etc/gitattributes

@christianbundy yes I already use that setting.

@dominic I think with the above you shouldn't have to deal with either the lockfile or the git diffs, are there any other inconveniences from lockfiles? If possible I'd love to use the lockfile default without negatively affecting anyone who wants to abstain from using them.

lock files!

yay package-lock.json in every library

yay npm-shrinkwrap.json in every interface (i didn't realize the new shrinkwraps were so simple! )

thanks @Christian Bundy for leading on this

Meta-discussion: how should issues like this be resolved?

Our deviation from the defaults makes it hard to debug things when they break (ssbc/ssb-db#261) and it doesn't feel like we're approaching consensus. I feel similarly about the StandardJS discussion (ssbc/ssb-server#591) which seemed to end in the same sort of deadlock.

I'm less focused on the individual problems and more interested in how we can resolve these sorts of problems in general. Anyone have thoughts on how this can work better?

BUMP

@dominic there's some gravel in @christianbundy's shoe - he's been doing some yard contributing the the JS stuff but has a blister. Would be good to stop and talk about some minor pain points so they don't become nasty ole big sores