leveldown Package Support #14

reconbot · 2016-01-14T23:33:35Z

I've come across my first popular package that can't cross compile.

[email protected]

While I'm not sure if this repo should be the support page for such problems long term, I think it's the perfect place to deal with this particular package. It's being used by @HipsterBrown via pouchdb.

The C code looks for some platform features that might be missing from our little tessel. I don't know enough to debug it.

../deps/leveldb/leveldb-1.18.0/port/atomic_pointer.h:211:2: error: #error Please implement AtomicPointer for this platform.
 #error Please implement AtomicPointer for this platform.

My google foo find me the file in question, a forum thread with possible fixes.

I took a step back for a moment and noticed they're not using node pre-gyp. They're using something called prebuild, and they claim to support the arm chipset.

I'm investigating further but I'd love help from anyone in the know.

The text was updated successfully, but these errors were encountered:

reconbot · 2016-01-14T23:49:19Z

I did a regular old npm install after setting all the cross compiling env vars. I had to manually make the tar of the package after that. I have the tar up on s3 for testing leveldown [email protected].

reconbot · 2016-01-14T23:49:28Z

@HipsterBrown is tribute to test it out

HipsterBrown · 2016-01-15T06:10:12Z

Smoke Test

My test project:

Setup:
mkdir tessel-test
t2 init
npm install --save pouchdb

Files:
index.js

var PouchDB = require('pouchdb');

var db = new PouchDB('./db');

db.put({
  "_id": "test",
  "desc": "This is a test doc."
});

db.info()
.then(function(info) {
  console.log(info);
});

console.log('This is a test.');

.tesselinclude

Included prebuild for compiling on the device.

node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/

Run project with t2 run index.js

CLI Patch

t2-cli/lib/tessel/deploy.js

The source of the binary was found in ~/.tessel/binaries/leveldown-1.4.3-Release/package/build/Release/leveldown.node instead of ~/.tessel/binaries/leveldown-1.4.3-Release/Release/leveldown.node. The fs.copySync also required a try/catch block to catch silent errors that were preventing continued execution of the deploy script.

actions.injectBinaryModules = function(globRoot, tempBundlePath) {
  return new Promise(resolve => {
    // For every binary module in use...
    binaryModulesUsed.forEach(details => {
    var sourceBinary = path.join(details.extractPath, 'package/build', 'Release', details.binName);
    var tempTargetModulePath = path.join(tempBundlePath, details.path);
    var tempTargetBinary = path.join(tempTargetModulePath, 'build/Release', details.binName);

    try {
      fs.copySync(sourceBinary, tempTargetBinary);
    } catch (err) {
      console.error(err);
    }
   });
   // All binary modules have been replaced, resolve.
   return resolve();
  });
};

Results:

After running with the patched CLI:

Nicks-MacBook-Pro:tessel-another-test hipsterbrown$ t2 run index.js --lan
INFO Looking for your Tessel...
INFO Connected to tessel-router.
INFO Writing project to RAM on tessel-router (8544.768 kB)...
INFO Deployed.
INFO Running index.js...
node: '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/build/Release/leveldown.node' is not an ELF executable for MIPS
This is a test.

That not ELF executable for MIPS error led me to this issue.

Then I tried compiling the binary on Tessel:

root@tessel-router:/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown# ./node_modules/prebuild/bin.js --install
prebuild info begin Prebuild version 2.8.1
prebuild info looking for local prebuild @ prebuilds/leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild info npm cache directory missing, creating it...
prebuild info looking for cached prebuild @ /root/.npm/_prebuilds/https-github.com-level-leveldown-releases-download-v1.4.3-leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild http request GET https://github.com/level/leveldown/releases/download/v1.4.3/leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild http 404 https://github.com/level/leveldown/releases/download/v1.4.3/leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild WARN install Prebuilt binaries for node version v4.2.1 are not available
prebuild info install We will now try to compile from source.
gyp info spawn /usr/bin/python2
gyp info spawn args [ '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/root/.node-gyp/4.2.1/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/root/.node-gyp/4.2.1',
gyp info spawn args   '-Dnode_gyp_dir=/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=node.lib',
gyp info spawn args   '-Dmodule_root_dir=/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.' ]
^C
root@tessel-router:/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown# Traceback (most recent call last):
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/gyp_main.py", line 16, in <module>
    sys.exit(gyp.script_main())
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 545, in script_main
    return main(sys.argv[1:])
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 538, in main
    return gyp_main(args)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 514, in gyp_main
    options.duplicate_basename_check)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 130, in Load
    params['parallel'], params['root_targets'])
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 2775, in Load
    variables, includes, depth, check, True)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 417, in LoadTargetBuildFile
    build_file_data, PHASE_EARLY, variables, build_file_path)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1292, in ProcessVariablesAndConditionsInDict
    build_file)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1307, in ProcessVariablesAndConditionsInList
    ProcessVariablesAndConditionsInDict(item, phase, variables, build_file)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1292, in ProcessVariablesAndConditionsInDict
    build_file)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1311, in ProcessVariablesAndConditionsInList
    expanded = ExpandVariables(item, phase, variables, build_file)
  File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 911, in ExpandVariables
    p_stdout, p_stderr = p.communicate('')
  File "/usr/lib/python2.7/subprocess.py", line 799, in communicate
    return self._communicate(input)
  File "/usr/lib/python2.7/subprocess.py", line 1409, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/usr/lib/python2.7/subprocess.py", line 1463, in _communicate_with_poll
    ready = poller.poll()
KeyboardInterrupt

This ended up hanging for too long. Still looking into that issue, but glad to get a successful deploy onto Tessel.

reconbot · 2016-01-15T14:50:11Z

I did wierd things when making the package.
On Jan 15, 2016 1:10 AM, "Nick Hehr" [email protected] wrote:

Smoke Test My test project:

Setup:
mkdir tessel-test
t2 init
npm install --save pouchdb

Files:
index.js

var PouchDB = require('pouchdb');
var db = new PouchDB('./db');
db.put({
"_id": "test",
"desc": "This is a test doc."
});
db.info()
.then(function(info) {
console.log(info);
});
console.log('This is a test.');

.tesselinclude

Included prebuild for compiling on the device.

node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/

Run project with t2 run index.js
CLI Patch

t2-cli/lib/tessel/deploy.js

The source of the binary was found in
~/.tessel/binaries/leveldown-1.4.3-Release/package/build/Release/leveldown.node
instead of
~/.tessel/binaries/leveldown-1.4.3-Release/Release/leveldown.node. The
fs.copySync also required a try/catch block to catch silent errors that
were preventing continued execution of the deploy script.

actions.injectBinaryModules = function(globRoot, tempBundlePath) {
return new Promise(resolve => {
// For every binary module in use...
binaryModulesUsed.forEach(details => {
var sourceBinary = path.join(details.extractPath, 'package/build', 'Release', details.binName);
var tempTargetModulePath = path.join(tempBundlePath, details.path);
var tempTargetBinary = path.join(tempTargetModulePath, 'build/Release', details.binName);
try {
  fs.copySync(sourceBinary, tempTargetBinary);
} catch (err) {
  console.error(err);
}
});
// All binary modules have been replaced, resolve.
return resolve();
});
};

Results:

After running with the patched CLI:

Nicks-MacBook-Pro:tessel-another-test hipsterbrown$ t2 run index.js --lan
INFO Looking for your Tessel...
INFO Connected to tessel-router.
INFO Writing project to RAM on tessel-router (8544.768 kB)...
INFO Deployed.
INFO Running index.js...
node: '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/build/Release/leveldown.node' is not an ELF executable for MIPS
This is a test.

That not ELF executable for MIPS error led me to this issue
nodejs/node#1440.

Then I tried compiling the binary on Tessel:

root@tessel-router:/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown# ./node_modules/prebuild/bin.js --install
prebuild info begin Prebuild version 2.8.1
prebuild info looking for local prebuild @ prebuilds/leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild info npm cache directory missing, creating it...
prebuild info looking for cached prebuild @ /root/.npm/_prebuilds/https-github.com-level-leveldown-releases-download-v1.4.3-leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild http request GET https://github.com/level/leveldown/releases/download/v1.4.3/leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild http 404 https://github.com/level/leveldown/releases/download/v1.4.3/leveldown-v1.4.3-node-v46-linux-mipsel.tar.gz
prebuild WARN install Prebuilt binaries for node version v4.2.1 are not available
prebuild info install We will now try to compile from source.
gyp info spawn /usr/bin/python2
gyp info spawn args [ '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args 'binding.gyp',
gyp info spawn args '-f',
gyp info spawn args 'make',
gyp info spawn args '-I',
gyp info spawn args '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/build/config.gypi',
gyp info spawn args '-I',
gyp info spawn args '/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/addon.gypi',
gyp info spawn args '-I',
gyp info spawn args '/root/.node-gyp/4.2.1/include/node/common.gypi',
gyp info spawn args '-Dlibrary=shared_library',
gyp info spawn args '-Dvisibility=default',
gyp info spawn args '-Dnode_root_dir=/root/.node-gyp/4.2.1',
gyp info spawn args '-Dnode_gyp_dir=/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp',
gyp info spawn args '-Dnode_lib_file=node.lib',
gyp info spawn args '-Dmodule_root_dir=/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown',
gyp info spawn args '--depth=.',
gyp info spawn args '--no-parallel',
gyp info spawn args '--generator-output',
gyp info spawn args 'build',
gyp info spawn args '-Goutput_dir=.' ]
^C
root@tessel-router:/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown# Traceback (most recent call last):
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/gyp_main.py", line 16, in
sys.exit(gyp.script_main())
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/init.py", line 545, in script_main
return main(sys.argv[1:])
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/init.py", line 538, in main
return gyp_main(args)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/init.py", line 514, in gyp_main
options.duplicate_basename_check)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/init.py", line 130, in Load
params['parallel'], params['root_targets'])
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 2775, in Load
variables, includes, depth, check, True)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 417, in LoadTargetBuildFile
build_file_data, PHASE_EARLY, variables, build_file_path)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1292, in ProcessVariablesAndConditionsInDict
build_file)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1307, in ProcessVariablesAndConditionsInList
ProcessVariablesAndConditionsInDict(item, phase, variables, build_file)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1292, in ProcessVariablesAndConditionsInDict
build_file)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 1311, in ProcessVariablesAndConditionsInList
expanded = ExpandVariables(item, phase, variables, build_file)
File "/tmp/remote-script/node_modules/pouchdb/node_modules/leveldown/node_modules/prebuild/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 911, in ExpandVariables
p_stdout, p_stderr = p.communicate('')
File "/usr/lib/python2.7/subprocess.py", line 799, in communicate
return self._communicate(input)
File "/usr/lib/python2.7/subprocess.py", line 1409, in _communicate
stdout, stderr = self._communicate_with_poll(input)
File "/usr/lib/python2.7/subprocess.py", line 1463, in _communicate_with_poll
ready = poller.poll()
KeyboardInterrupt

This ended up hanging for too long. Still looking into that issue, but
glad to get a successful deploy onto Tessel.

—
Reply to this email directly or view it on GitHub
#14 (comment).

johnnyman727 · 2016-01-18T18:15:43Z

@reconbot not sure if I can be helpful here but I'd like to understand the issue(s). Is there a problem with how T2 handles prebuild and an issue with the underlying atomicpointer library? (Along with the two CLI issues @HipsterBrown caught)

reconbot · 2016-01-19T04:42:58Z

I think our issue is with prebuild and not atomicpointer. Give it a whirl
and try to build the leveldown package and see what happens.

On Mon, Jan 18, 2016 at 11:15 AM, Jon [email protected] wrote:

@reconbot https://github.com/reconbot not sure if I can be helpful here
but I'd like to understand the issue(s). Is there a problem with how T2
handles prebuild and an issue with the underlying atomicpointer library?
(Along with the two CLI issues @HipsterBrown
https://github.com/HipsterBrown caught)

—
Reply to this email directly or view it on GitHub
#14 (comment).

Francis Gulotta
Director of Web Connected Devices
Mobile: 917-805-3079

johnnyman727 · 2016-01-21T01:08:14Z

Ah I see - well I probably won't be much help on this one. @tcr might have some advice.

nolanlawson · 2016-06-07T20:17:24Z

FWIW I also tried the sqlite3 package (PouchDB's other Node-supported backing database), but no luck so far: #27

reconbot mentioned this issue Jan 15, 2016

Cross compiling for ARM/MIPS prebuild/prebuild#69

Closed

This was referenced Jan 15, 2016

copySync errors swallowed, hanging deploy tessel/t2-cli#524

Closed

Binary not found in details.extractPath, 'Release', details.binName tessel/t2-cli#526

Closed

nolanlawson mentioned this issue Jun 7, 2016

Cannot compile node-sqlite3 #27

Open

reconbot changed the title ~~Package Support~~ leveldown Package Support Jun 13, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

leveldown Package Support #14

leveldown Package Support #14

reconbot commented Jan 14, 2016

reconbot commented Jan 14, 2016

reconbot commented Jan 14, 2016

HipsterBrown commented Jan 15, 2016

reconbot commented Jan 15, 2016

johnnyman727 commented Jan 18, 2016

reconbot commented Jan 19, 2016

johnnyman727 commented Jan 21, 2016

nolanlawson commented Jun 7, 2016

leveldown Package Support #14

leveldown Package Support #14

Comments

reconbot commented Jan 14, 2016

reconbot commented Jan 14, 2016

reconbot commented Jan 14, 2016

HipsterBrown commented Jan 15, 2016

Smoke Test

My test project:

CLI Patch

Results:

reconbot commented Jan 15, 2016

johnnyman727 commented Jan 18, 2016

reconbot commented Jan 19, 2016

johnnyman727 commented Jan 21, 2016

nolanlawson commented Jun 7, 2016