Keengreeper

Keep git-hosted NodeJS projects up-to-date - shrinkwrap-capable.

AKA: GreenKeeper replacement. Following on the forced migration to GreenKeeper2,
and the late annouce that we would now be charged: here is a rudimentary drop-in
replacement, that actually handles shrinkwrap, and may also be used raising
alerts based on Snyk security advisories.

Dependencies

bash (installing & running NVM)
curl (querying NPMJS & Snyk, installing NVM, querying CircleCI, notifying
slack)
git, runtime user should have username & email configured
ssh key pair, setting up passwordless access to your repositories

Introducing KeenGreeper

The configure script would allow you to create and manage a
leightweight database listing the repositories we would track, a cache of the
NodeJS dependencies involved and their last known versions, as well as a list of
modules whose updates we should ignore.
Running this pseudo-shell, type h or help to display the help menu.

The updateModules script should then iterate on previously registered
repositories, making sure they’re working on the staging branch (if such exists,
master otherwise), pulling the lastest changes from git upstream, looking for
dependencies listed in the corresponding package.json, checking for known
updates. Whenever some new versions are detected, a branch is created,
your package.json gets updated. If a node_shrinkwrap folder exists, its
content would be updated as well. Eventually these new branches are pushed to
upstream for further verification (hopefully: CI).

Then, the updateVulnerabilities script may be used to iterate on previously
cached NodeJS modules, looking for potential vulnerabilities.

The expireCaches drops references to dependencies from your NodeJS modules &
Snyk caches, when these are no longer relevant to your repositories.

The checkVulnerabilities checks your dependencies against Snyk advisories,
and eventually sends Slack and/or mail notifications.

The mergeBranches checks for previously-pushed patch branches, queries
CircleCI and may merge successfully-tested patches, eventually cleaning up
previously-pushed branches.

Setup

We first need to set some variables. Look at env.sample for an example, you
would have to define:

DBDIR, the folder we would keep our internal configurations & caches into
CACHETTL, the amount of seconds a record from npmjs.org should be kept
CIRCLE_TOKEN, optional token querying CircleCI for tests results, assuming
some of your repositories could be private
LOGDIR, the folder we would store npm install and shrinkwrap logs into
MAIL_RCPT, the recipients of mail alerts upon matching a vulnerable modules
SLACK_HOOK_URL, upon matching vulnerable modules, send Slack notification
SNYKTTL, the amount of seconds a record from Snyk should be kept
TMPDIR, the folder we will be preparing our new commits into
WORKDIR, the folder we would keep cloned repositories into

Having set the proper values, make sure to install that file as
/etc/keengreeper.conf, to have it loaded automatically - otherwise, make sure
to source that environment file before running any of the enclosed scripts. Also
make sure the directories involved do exist and are writeable by the user that
will run our scripts.

Then, we can start registering repositories:

git/keengreeper$ ./configure
WARNING: nodejs-update-db absent, creating empty dataset
<$ type h or help for usage, q or quit to exit
$> a PeerioTechnologies/peerio-inferno
Cloning into 'peerio-inferno'...
remote: Counting objects: 3806, done.
remote: Compressing objects: 100% (72/72), done.
remote: Total 3806 (delta 18), reused 0 (delta 0), pack-reused 3734
Receiving objects: 100% (3806/3806), 56.82 MiB | 11.56 MiB/s, done.
Resolving deltas: 100% (1758/1758), done.
Checking connectivity... done.
<add$ NOTICE: added PeerioTechnologies/peerio-inferno
$> add git@mygitlab.example.com:DevOps/awesome-project
Cloning into 'peerio-shark'...
remote: Counting objects: 6732, done.
remote: Total 6732 (delta 0), reused 0 (delta 0), pack-reused 6731
Receiving objects: 100% (6732/6732), 66.28 MiB | 11.15 MiB/s, done.
Resolving deltas: 100% (3696/3696), done.
Checking connectivity... done.
<add$ NOTICE: added git@mygitlab.example.com:DevOps/awesome-project
$> quit

Note that we would be assuming the user running these scripts can already
passwordlessly clone and write to the repositories you would add. Using an SSH
key is recommended. Running these scripts from some un-attended station:
creating a separate git user with limited privileges is encourraged.

You may configure a couple modules that shouldn’t be taken into consideration
by our update process, whatever the reasons:

git/keengreeper$ ./configure
$> i mailparser
<add$ NOTICE: added mailparser
$> ign basho-riak-client
<add$ NOTICE: added basho-riak-client
$> ignore sinon
<add$ NOTICE: added sinon
$> di mailparser
<del$ NOTICE: removed mailparser from ignored dependencies
$> li
basho-riak-client
sinon
$> quit

Finally, you can use the main script updating your projects dependencies:

git/keengreeper$ ./updateModules
refreshing npmjs local cache
processing peerio-server
skipping peerio-server::basho-riak-client - dependency registered to exclude file
skipping peerio-server::sinon - dependency registered to exclude file
processing desktop-update-proxy
staging q@desktop-update-proxy 1.4.1 -> 1.5.0
building staged changes for desktop-update-proxy
[chaos-q1.5.0 5c6edf6] chore(package): bump q to 1.5.0
 1 file changed, 1 insertion(+), 1 deletion(-)
done

Being sure our magic works, you may consider setting up some cron job:

(
    crontab -l
    echo "30 */2 * * * `pwd`/updateModules >>log/cron.log 2>&1"
    echo "45 */6 * * * `pwd`/updateVulnerabilities >>log/snyk-cron.log 2>&1"
    echo "15   0 * * * `pwd`/expireCaches >>log/cache-cron.log 2>&1"
    echo " 0  22 * * * `pwd`/checkVulnerabilities >>log/vuln-cron.log 2>&1"
    echo "20 */2 * * * `pwd`/mergeBranches >>log/merge-cron.log 2>&1"
) | crontab -