New Issue

arm64v8 builds fail due to outdated dependency #346

Closed

opened by mcrosson · 17 comments

mcrosson commented (Migrated from github.com)

Plume version (and/or commit): 61b6ceed92

During compile on arm64v8 the following error surfaces during the final step (looks like linking). From what I can tell rust-crypto is no longer maintained and won't be receiving a fix (see https://github.com/DaGenix/rust-crypto/issues/383 for reference).

I'm not familiar with the rust toolchain and this seems like Plume or one of it's dependencies has a dependency on the legacy rust-crypto library. Is there a way to track this down and/or update the code to avoid rust-crypto popping up in the dependency graph?

I bring this up because a Raspberry Pi 3, Orange Pi, Scaleway arm VPSs and Amazon arm VPSs are all cheap, easy to obtain and can be used for self hosting yet are all arm64v8 devices/servers.

warning: struct is never constructed: `Write`                                                                                                                                                                                          
  --> src/api/authorization.rs:19:1                                                                                                                                                                                                    
   |                                                                                                                                                                                                                                   
19 | pub struct Write;                                                                                                                                                                                                                 
   | ^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                 
   |                                                                                                                                                                                                                                   
   = note: #[warn(dead_code)] on by default                                                                                                                                                                                            
                                                                                                                                                                                                                                       
error: linking with `cc` failed: exit code: 1                                                                                                                                                                                          
  |                                             
[snip]



  = note: /app/target/debug/deps/libcrypto-0e2d8e020a3b8dff.rlib(crypto-0e2d8e020a3b8dff.crypto.277lu5c6-cgu.12.rcgu.o): In function `crypto::util::fixed_time_eq':
          /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rust-crypto-0.2.36/src/util.rs:52: undefined reference to `rust_crypto_util_fixed_time_eq_asm'
          collect2: error: ld returned 1 exit status
          

error: aborting due to previous error

error: failed to compile `plume v0.2.0 (/app)`, intermediate artifacts can be found at `/app/target`

Plume version (and/or commit): 61b6ceed92ef838a085da6eb66f87ab1de65a520 During compile on arm64v8 the following error surfaces during the final step (looks like linking). From what I can tell rust-crypto is no longer maintained and won't be receiving a fix (see https://github.com/DaGenix/rust-crypto/issues/383 for reference). I'm not familiar with the rust toolchain and this seems like Plume or one of it's dependencies has a dependency on the legacy rust-crypto library. Is there a way to track this down and/or update the code to avoid rust-crypto popping up in the dependency graph? I bring this up because a Raspberry Pi 3, Orange Pi, Scaleway arm VPSs and Amazon arm VPSs are all cheap, easy to obtain and can be used for self hosting yet are all arm64v8 devices/servers. ``` warning: struct is never constructed: `Write` --> src/api/authorization.rs:19:1 | 19 | pub struct Write; | ^^^^^^^^^^^^^^^^^ | = note: #[warn(dead_code)] on by default error: linking with `cc` failed: exit code: 1 | [snip] = note: /app/target/debug/deps/libcrypto-0e2d8e020a3b8dff.rlib(crypto-0e2d8e020a3b8dff.crypto.277lu5c6-cgu.12.rcgu.o): In function `crypto::util::fixed_time_eq': /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rust-crypto-0.2.36/src/util.rs:52: undefined reference to `rust_crypto_util_fixed_time_eq_asm' collect2: error: ld returned 1 exit status error: aborting due to previous error error: failed to compile `plume v0.2.0 (/app)`, intermediate artifacts can be found at `/app/target` ```

elegaanz commented (Migrated from github.com)

This dependency is here because of csrf, that is here because of rocket_csrf, but csrf doesn't seem maintained anymore (last update was one year ago), so I don't know what we can do… forking csrf to replace the rust-crypto dependency with something else would probably require a lot of work. Otherwise, we can wait for Rocket to implement CSRF directly (see https://github.com/SergioBenitez/Rocket/issues/14) and hope that they wont use csrf…

This dependency is here because of `csrf`, that is here because of `rocket_csrf`, but `csrf` doesn't seem maintained anymore (last update was one year ago), so I don't know what we can do… forking `csrf` to replace the `rust-crypto` dependency with something else would probably require a lot of work. Otherwise, we can wait for Rocket to implement CSRF directly (see https://github.com/SergioBenitez/Rocket/issues/14) and hope that they wont use `csrf`…

mcrosson commented (Migrated from github.com)

@BaptisteGelez Could we fork csrf and rust-crypto with the armv8 patch from my above reference as a stop-gap?

I'm not sure of the overall cost/benefit of such a move is though. Especially when it comes to tuning underlying deps "creatively" during build.

@BaptisteGelez Could we fork csrf and rust-crypto with the armv8 patch from my above reference as a stop-gap? I'm not sure of the overall cost/benefit of such a move is though. Especially when it comes to tuning underlying deps "creatively" during build.

trinity-1686a commented

Owner

@BaptisteGelez I think at some point we should test compilation for this architecture on travis (although it can't run tests obviously). Tantivy is doing so, so we could copy part of their travis.yml

mcrosson commented (Migrated from github.com)

@fdb-hiroshima If you'd like I'm more than willing to donate an Orange Pi or Raspberry Pi to the project for compilation testing...

In the mean time I do have a build job in Jenkins for a project I maintain for arm32v7, arm64v8 and x86-64 to build the Docker images.

@fdb-hiroshima If you'd like I'm more than willing to donate an Orange Pi or Raspberry Pi to the project for compilation testing... In the mean time I do have a build job in Jenkins for a project I maintain for arm32v7, arm64v8 and x86-64 to build the Docker images.

ProgVal commented (Migrated from github.com)

although it can't run tests obviously

There's qemu-aarch64-static. It will probably be very slow, though

> although it can't run tests obviously There's qemu-aarch64-static. It will probably be very slow, though

👍 1

trinity-1686a commented

Owner

@mcrosson I have multiples Pis (no Orange, but both Bananas and Raspberries) however I believe this might not be a good idea for one of us to host that stuff. I feel like it's giving to much importance to a single people, who might for whatever reason not be available when there are issue only they can fix. In the meantime I'll try to change rocket_csrf dependencies to require a crypto library that is actually maintained.
@ProgVal I do think so too, but we could actually test and see, however we will still need to cross-compile. Compiling inside an emulator is damn slow, and rust is not known to be of the quick type on compilation

@mcrosson I have multiples Pis (no Orange, but both Bananas and Raspberries) however I believe this might not be a good idea for one of us to host that stuff. I feel like it's giving to much importance to a single people, who might for whatever reason not be available when there are issue only they can fix. In the meantime I'll try to change rocket_csrf dependencies to require a crypto library that is actually maintained. @ProgVal I do think so too, but we could actually test and see, however we will still need to cross-compile. Compiling inside an emulator is damn slow, and rust is not known to be of the quick type on compilation

👍 2

ProgVal commented (Migrated from github.com)

however we will still need to cross-compile. Compiling inside an emulator is damn slow

You could first cross-compile with a native compiler (using cargo build --target XXX), then run qemu-aarch64-static cargo test

> however we will still need to cross-compile. Compiling inside an emulator is damn slow You could first cross-compile with a native compiler (using `cargo build --target XXX`), then run `qemu-aarch64-static cargo test`

👍 2

mcrosson commented (Migrated from github.com)

@fdb-hiroshima I totally understand re 1 person

@ProgVal / @fdb-hiroshima I have done the cross compilation thing with qemu before and... it's brutally slow even on a semi-modern vintage xeon.

@fdb-hiroshima I totally understand re 1 person @ProgVal / @fdb-hiroshima I have done the cross compilation thing with qemu before and... it's brutally slow even on a semi-modern vintage xeon.

mcrosson commented (Migrated from github.com)

@fdb-hiroshima I just found out (and verified) that the Raspberry Pi B is arm32v7 while the Raspberry Pi B+ is arm64v8 (unverified).

I'm going to be working with another dev on verifying everything hardware wise in rPi B[+] land. It might be good to both arm32 and 64 while the various board makers are releasing both platforms...

@fdb-hiroshima I just found out (and verified) that the Raspberry Pi B is arm32v7 while the Raspberry Pi B+ is arm64v8 (unverified). I'm going to be working with another dev on verifying everything hardware wise in rPi B[+] land. It might be good to both arm32 and 64 while the various board makers are releasing both platforms...

👍 1

trinity-1686a commented

Owner

I've modified rocket_csrf so it won't require rust-crypto, however this mean I now use some cryptographic primitives directly instead of someone else doing it for me, so I would prefer if someone else, preferably with some knowledge about cryptography could review it here (just to be clear this isn't the first time I play with crypto, but as this is a particularly sensible field I never feel safe doing so, nor should I).

I've modified `rocket_csrf` so it won't require `rust-crypto`, however this mean I now use some cryptographic primitives directly instead of someone else doing it for me, so I would prefer if someone else, preferably with some knowledge about cryptography could review it [here](https://github.com/fdb-hiroshima/rocket_csrf/blob/9eb7b3f98baae4d833b34be15012d300884a1cab/src/crypto.rs) (just to be clear this isn't the first time I play with crypto, but as this is a particularly sensible field I never feel safe doing so, nor should I).

👍 1

ProgVal commented (Migrated from github.com)

let token_ok = &token.token == &cookie.token;

Use constant_time::verify_slices_are_equal instead. There's probably no way to run a timing attack via a CSRF, but let's not give it a chance.

Other than that, it's ok as far as I can tell.

``` let token_ok = &token.token == &cookie.token; ``` Use `constant_time::verify_slices_are_equal` instead. There's probably no way to run a timing attack via a CSRF, but let's not give it a chance. Other than that, it's ok as far as I can tell.

trinity-1686a commented

Owner

Thanks @ProgVal , I've updated accordingly. If nobody have more advice, I'll change Plume's dependency version to this and it should compile on armv8 by Thursday evening

mcrosson commented (Migrated from github.com)

@fdb-hiroshima i think i figured out how to update the git version used by the build process for docker.

I'll report back if it's a clean build

@fdb-hiroshima i *think* i figured out how to update the git version used by the build process for docker. I'll report back if it's a clean build

mcrosson commented (Migrated from github.com)

@fdb-hiroshima This is what I got when trying to run a Plume build just now...

warning: Using `cargo install` to install the binaries for the package in current working directory is deprecated, use `cargo install --path .` instead. Use `cargo build` if you want to simply build the package.
  Installing plume v0.2.0 (/app)
    Updating git repository `https://github.com/fdb-hiroshima/rocket_csrf`
    Updating crates.io index
error: failed to compile `plume v0.2.0 (/app)`, intermediate artifacts can be found at `/app/target`

Caused by:
  failed to select a version for `ring`.
    ... required by package `cookie v0.11.0`
    ... which is depended on by `rocket_http v0.4.0`
    ... which is depended on by `rocket v0.4.0`
    ... which is depended on by `plume-common v0.2.0 (/app/plume-common)`
    ... which is depended on by `plume v0.2.0 (/app)`
versions that meet the requirements `= 0.13.2` are: 0.13.2

the package `ring` links to the native library `ring-asm`, but it conflicts with a previous package which links to `ring-asm` as well:
package `ring v0.13.5`
    ... which is depended on by `rocket_csrf v0.1.0 (https://github.com/fdb-hiroshima/rocket_csrf?rev=4a72ea2ec716cb0b26188fb00bccf2ef7d1e031c#4a72ea2e)`
    ... which is depended on by `plume v0.2.0 (/app)`

failed to select a version for `ring` which could resolve this conflict
The command '/bin/sh -c cargo install --force --debug --no-default-features --features postgres' returned a non-zero code: 101

Local changes

diff --git a/Cargo.toml b/Cargo.toml
index 79e6294..992abab 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -59,7 +59,7 @@ path = "plume-models"

 [dependencies.rocket_csrf]
 git = "https://github.com/fdb-hiroshima/rocket_csrf"
-rev = "717fad53cfd2ee5cbee5b4571f6190644f9dddd7"
+rev = "4a72ea2ec716cb0b26188fb00bccf2ef7d1e031c"

 [build-dependencies]
 ructe = "0.5.4"

@fdb-hiroshima This is what I got when trying to run a Plume build just now... ``` warning: Using `cargo install` to install the binaries for the package in current working directory is deprecated, use `cargo install --path .` instead. Use `cargo build` if you want to simply build the package. Installing plume v0.2.0 (/app) Updating git repository `https://github.com/fdb-hiroshima/rocket_csrf` Updating crates.io index error: failed to compile `plume v0.2.0 (/app)`, intermediate artifacts can be found at `/app/target` Caused by: failed to select a version for `ring`. ... required by package `cookie v0.11.0` ... which is depended on by `rocket_http v0.4.0` ... which is depended on by `rocket v0.4.0` ... which is depended on by `plume-common v0.2.0 (/app/plume-common)` ... which is depended on by `plume v0.2.0 (/app)` versions that meet the requirements `= 0.13.2` are: 0.13.2 the package `ring` links to the native library `ring-asm`, but it conflicts with a previous package which links to `ring-asm` as well: package `ring v0.13.5` ... which is depended on by `rocket_csrf v0.1.0 (https://github.com/fdb-hiroshima/rocket_csrf?rev=4a72ea2ec716cb0b26188fb00bccf2ef7d1e031c#4a72ea2e)` ... which is depended on by `plume v0.2.0 (/app)` failed to select a version for `ring` which could resolve this conflict The command '/bin/sh -c cargo install --force --debug --no-default-features --features postgres' returned a non-zero code: 101 ``` Local changes ``` diff --git a/Cargo.toml b/Cargo.toml index 79e6294..992abab 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -59,7 +59,7 @@ path = "plume-models" [dependencies.rocket_csrf] git = "https://github.com/fdb-hiroshima/rocket_csrf" -rev = "717fad53cfd2ee5cbee5b4571f6190644f9dddd7" +rev = "4a72ea2ec716cb0b26188fb00bccf2ef7d1e031c" [build-dependencies] ructe = "0.5.4" ```

trinity-1686a commented

Owner

Yes, you can't link to 2 versions of ring at the same time. You can run cargo update -p ring and it will upgrade Cargo.lock to use ring 0.13.5 for cookie 0.11.0, that way it will compile fine (I will do it myself when I commit the version change)

Yes, you can't link to 2 versions of ring at the same time. You can run `cargo update -p ring` and it will upgrade Cargo.lock to use ring 0.13.5 for cookie 0.11.0, that way it will compile fine (I will do it myself when I commit the version change)

👍 1

mcrosson commented (Migrated from github.com)

@fdb-hiroshima I'll run that as part of the docker build and run a new test.

mcrosson commented (Migrated from github.com)

@fdb-hiroshima Plume just built for me on arm64v8.

root@ubuntu:~# docker run --rm -it arm64v8/plume:2870642cfad4bee3d6d9b76852f231b
199bebc17 plm help 
Plume CLI 0.2.0
Collection of tools to manage your Plume instance.

USAGE:
    plm [SUBCOMMAND]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

SUBCOMMANDS:
    help        Prints this message or the help of the given subcommand(s)
    instance    Manage instances
    search      Manage search index
    users       Manage users

@fdb-hiroshima Plume just built for me on arm64v8. ``` root@ubuntu:~# docker run --rm -it arm64v8/plume:2870642cfad4bee3d6d9b76852f231b 199bebc17 plm help Plume CLI 0.2.0 Collection of tools to manage your Plume instance. USAGE: plm [SUBCOMMAND] FLAGS: -h, --help Prints help information -V, --version Prints version information SUBCOMMANDS: help Prints this message or the help of the given subcommand(s) instance Manage instances search Manage search index users Manage users ```

🎉 2

No Milestone

No Assignees

2 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Loading…

There is no content yet.

Labels Milestones

arm64v8 builds fail due to outdated dependency #346