module system

rvagg:

I don't feel strongly either way about this. I'm not a strict minimalist but like Dominic I tend to use batch() programatically so a chaining API is less helpful (but I may use it).

What we really need is a proper plugin system for Node so a project like LevelUP can load optional plugins that may be installed in the NODE_MODULES path(s) and do things like expose the main LevelUP.prototype to plugins that may want to augment it. Then we could say something on our README like: "If you npm install levelup-batch, LevelUP will make the chaining batch() API available. And levelup-batch can monkey-patch LevelUP.prototype.batch to provide chaining.

I need something like this for Ender too so maybe I'll actually build a generic plugin helper system some day..

rvagg:

Re plugins, Grunt and DocPad do something similar already but certainly not in a modular way that you can pull out and re-use, we need something that smaller projects like LevelUP can easily include. We should collaborate on it so we get the API right!

it should detect collisions
it should load all plugins from a specified folder
it should extend the prototype

Will it do the extending automatically or should we say db.use('batch', require('cbatch'))

I'm thinking that package.json specifies the extension 'types' that the main project exports. So in the case of LevelUP we may have an extension 'type' of "prototype" and if the package.json includes that then LevelUP needs to expose it to the plugin system somehow which then passes it in a predefined manner to the plugin(s) that need it.

I think extending should be done automatically. How about:

db.require('cbatch')

or perhaps .use() is common practice?

db.use('cbatch')

So, levelup-batch has a "levelup": { "pluginType": [ "prototype" ] } in package.json, then the plugin system scans NODE_MODULES and finds this module with a "levelup" descriptor in package.json and adds it to the list of plugins. Then LevelUP somewhere does something like plugins.expose('prototype', LevelUP.prototype) (this is simplistic, you'd want to expose other stuff that wouldn't fit nicely into this pattern). and then somewhere else can do 'plugins.load(callback) which does all the exposing/extending stuff by working through all the plugins it found.

Make sense? It does in my head at least.

@ralphtheninja

@ralphtheninja I want something that doesn't require LevelUP to be aware of the various plugins that it can load; that way you can write a plugin without LevelUP-core even knowing about it, you could even write a completely private one that never sees the light of day outside your company.

@rvagg

@rvagg Aye, makes sense. It feels like a plugin pattern :)

I think we could scratch the "pluginType" for now and just assume one way of doing it. Pick the simplest use case and make it work, then introduce other ways of extending.

automatic plugins will come back and bite us on the ass.
in my levelup stuff (see all my modules that are prefixed level-*) i initially used a db.use(plugin) pattern
but then abandoned that, and just went with plugin(db) the plugin is responsible for checking that it's not already attached, and to make sure that it only minimally alters the db object - I stuck with only adding one patch that has the same name as the plugin itself.

This method is rather ugly, but at least it's simple.

So, I built a bunch of stuff with this pattern, some plugins just patched the db object, but others actually inserted data into the database. This meant prefixing the keys of inserted data.

I did that enough, that now I'm thinking that a much better approach would be to abstract that all away,
go db2 = db.namespace('PREFIX') to get another db, and then do something like
map(db1, db2, function map(k, v, emit) { emit(k, 1) }) etc...

this could be much more flexible, and also cleaner - but there is a bit of thinking on how to get everything working together. Most of my stuff is based at some point on hooking into a put and turning it into a batch that inserts a job, or other keys atomically. it would be nice to cleanly isolate these sections, but i'd need also need to be able to insert into several sections atomically to get consistency which is the objective at the end of the day.

There's no need for a plugin systems. We already have commonJS

var myPlugin = require("level-plugin")

myPlugin(db, "do shit", function (err, result) {

})

Functions are really, really simple and work universally.

Now if you want a plugin system because you want to hook into put / del / batch etc, then we should have a single module that uses hooks and everyone uses that.

As for detecting collisions. We already have a namespaces enforced by npm. It'll be easier to use those.

@Raynos

@Raynos calm down and clearly explain what you think, and why you think it.
I know you, so I know how you really mean it, but carrying on like this does not make for a pleasant and constructive discussion.

I'll be sensible.

Why do we need a plugin system? What benefit does it give us over just using require ? Can we enumerate the use-cases for plugins and the scenarios where they are useful, as I can't think of any.

we have a collection of modules, which are extensions to leveldb - they are firmly coupled, because they all interact with levelup some how.

We want them to be as loosly coupled as possible, so that it's possible to iterate on the levelup implementation without breaking them, if they depend on incidental implementation details of levelup then they will be fragile.

So, we already have a partial list here https://github.com/rvagg/node-levelup/wiki/Modules

And there are more that we want to build, but havn't gotten around to implementing yet. I think what this discussion really wants to be about is 'what is the interface that extensions should interact with?'.

certainly speaking for myself, getting the plugins that I wrote correct was quite difficult, and I think there is a lot of room for improvement here.

@dominictarr

@dominictarr the interface is the same as the interface for levelup.

If we make a breaking change in levelup then

a) we need to update all apps that use levelup to use the new API
b) we need to update all levelup modules to use the new API

We should be using peer-dependencies in our levelup modules to lock the version.

I agree that making a breaking improvement in levelup is a pain for module owners but that's how it works. This the downside of modular javascript, if you have many intrarelated modules then you need to update them all in one batch. I don't know any answer for this.

I have had issues with it with Raynos/signal-channel, Raynos/read-write-stream and @Gozala has issues with it for gozala/reducers

this wiki page outlines what I want to use levelup plugins for... please comment if you have questions, or want me to clarify something

Why was the following api chosen

var hooks = require('level-hooks')
hooks(db)
db.hooks.stuff()

instead of

var hooks = require('level-hooks')(db)
hooks.stuff()

Thanks for the writeup. Have the same question as Raynos.

On 30 January 2013 02:43, Raynos notifications@github.com wrote:

Why was the following api chosen

var hooks = require('level-hooks')hooks(db)db.hooks.stuff()

instead of

var hooks = require('level-hooks')(db)hooks.stuff()

—
Reply to this email directly or view it on GitHubhttps://github.com/rvagg/node-levelup/issues/68#issuecomment-12869576.

@Raynos

Because I want to ensure that there is only one instance of hooks.
I know it's ugly, but since it actually inserts data,
(with a prefix, but basically into a global namespace)
then you basically have monkey-patched the data, inside the database.

Doing it some overly clever way, like weak maps, or something will probably lead to data collisions or something.

However, if there was a better way of separating prefixed sections - like a built in namespace thing, and here the database object would STILL need to keep track of what prefixes things are using. - there may be a way to do it as you describe.

So, I agree with your objections @Raynos and I'd never normally do it like this, I only did it this way this time because I couldn't not monkey-patch the data, so this was was simpler.

If there was a really clean way to partition the data, then I could remove that stuff.

@dominictarr

@dominictarr i.e. this would be a problem

var hooks1 = hooks(db)
var hooks2 = hooks(db)

hooks1.pre(mutate1)
hooks2.pre(mutate2)

Because the intercepted put would be persisted to the db twice?

One thing you can do is memoize hooks to return the same thing given a db but that would still cause dedup problems if there are TWO seperate hooks modules.

@dominictarr what about a compromise?

function hooks(db) {
  if (db.__hooks__exists) {
    return db.__hooks__exists
  }

  ... 

  db.__hooks__exists = hooksThing
  return hooksThing
}

That way you keep the invariant but you can have the nicer API of just returning the thing.

This is only worthwhile if most other plugins don't need to put their methods as a namespace on the db and can just return themself.

@Raynos

@Raynos that is exactly what I am doing, except without the __ prefix. I didn't try to hide it, because I wanted it to be apparent what was going on, and so that people would notice. Looks like it worked! :)

hooks is an exception, because it doesn't actually insert any data into the database, it just patches the db object.

level-trigger is a better example, the put wouldn't be persisted twice, because hooks doesn't handle that, db still does. If two triggers where created with the same prefix, but different rules... things will break, because the order of parallel inserts is not reliable (i've tested this)

A plugin needs to get the same prefix after the process restarts, but a different prefix than any other plugin.

so in summary, I think that a leveldb plugin system needs

reliably separated areas of data.
pre/post hooks on put, del, batch

👍 to reliably seperated areas of data.

Hooks are needed but may work as a userland module

well, hooks monkey patch put, del, and batch. And depends on fairly tight implementation details, that is why it might be good to have it tied into levelup. Especially now that we have leveldown, and if you want a super light weight binding, you can just use that!

isaacs/npm#1400

I'm using hooks right now, they are awesome and im totally happy with them being a module.

The situation with levelup has changed quite a lot since we started this thread, since we now have levelDown.
This is a new thing, prehaps, because we have an inner-core, and and outer-core.

I think this relaxes some of our initial concerns - because we have levelDown for a minimum binding, and levelup for convenience.

It may be a good idea to bundle everything needed to create a stable platform for extending the database - but that will have to wait until we've really figured out what the generalized foundations are.

Navigation Menu

Repository navigation

module system #52

15 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

module system #52

Description

Activity

juliangruber commented on Jan 28, 2013

rvagg commented on Jan 28, 2013

ralphtheninja commented on Jan 28, 2013

rvagg commented on Jan 28, 2013

rvagg commented on Jan 28, 2013

ralphtheninja commented on Jan 28, 2013

ralphtheninja commented on Jan 28, 2013

dominictarr commented on Jan 28, 2013

Raynos commented on Jan 28, 2013

Raynos commented on Jan 28, 2013

dominictarr commented on Jan 28, 2013

Raynos commented on Jan 28, 2013

dominictarr commented on Jan 28, 2013

Raynos commented on Jan 28, 2013

15 remaining items

dominictarr commented on Jan 30, 2013

Raynos commented on Jan 30, 2013

ralphtheninja commented on Jan 30, 2013

dominictarr commented on Jan 30, 2013

Raynos commented on Jan 30, 2013

dominictarr commented on Jan 30, 2013

Raynos commented on Jan 30, 2013

dominictarr commented on Jan 30, 2013

ralphtheninja commented on Jan 30, 2013

heapwolf commented on Feb 16, 2013

dominictarr commented on Feb 16, 2013

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions