Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

typescript bindings updates, cuda support #2403

Open
wants to merge 28 commits into
base: main
Choose a base branch
from
Open

typescript bindings updates, cuda support #2403

wants to merge 28 commits into from

Conversation

iimez
Copy link
Collaborator

@iimez iimez commented Jun 4, 2024 •
edited
Loading

Describe your changes

  • Changed the build scripts and docs a bit so its easier to build the bindings from source.
  • Deleted some outdated comments from the gyp config, and removed the deprecated nCtx on completions.
  • Update the build_unix script to produce so files with the correct build type.
  • Changed CI to call cmake with Release build type.
  • Added support for streamed token decoding, following the python bindings implementation
  • Updated the bindings to expose additional LLModel:load and LLModel:initGpu methods so its easier to copy the loading and backend-picking logic of the CUDA Backend PR. I also renamed a few things for consistency.
  • Include nThreads in the LoadModelOptions, previously it had to be configured via setThreadCount.
  • Added nGpuLayers as alias for ngl to the LoadModelOptions.
  • Updated types and examples accordingly.
  • I added a test suite that does actual inference and loads a real model on both cpu and gpu with minimal resource usage. And moved the existing tests into a separate test:ci script. Can switch CI config back to yarn test if inference in CI is possible.
  • Updated deps and verified that the fallback node-addon build via node-gyp-build is still working. Changed the build setup a bit to include a assert-backend-sources.js script that facilitates easier testing of the fallback during development. This also allows us to get rid of the second bindings.gyp config.

Breaking changes

  • The CompletionOptions onReponseToken(tokenId, text) callback has been changed to onResponseTokens({ tokenIds, text }) (Note the plural.). This is related to the streamed emoji decoding / multi token character issue.

Issue ticket number and link

#2326

@cebtenzzre cebtenzzre mentioned this pull request Jun 4, 2024
Open
4 tasks
@cebtenzzre cebtenzzre mentioned this pull request Jul 17, 2024
Closed
@cebtenzzre
Copy link
Member

build-bindings-backend-windows-msvc (now just called build-bindings-backend-windows) uses a deprecated image and will need to be updated to use the current tag (or the windows orb). See #2685.

jacoobes and others added 10 commits August 24, 2024 04:17
@jacoobes @iimez
update typings
68f4097 
Signed-off-by: jacob <[email protected]>

update typings

Signed-off-by: jacob <[email protected]>
Signed-off-by: limez <[email protected]>
@jacoobes @iimez
readme,mspell
de0e096 
Signed-off-by: jacob <[email protected]>
@iimez
cuda/backend logic changes + name napi methods like their js counterp…
db6713f 
…arts

Signed-off-by: limez <[email protected]>
@iimez
convert llmodel example into a test, separate test suite that can run…
35758f0 
… in ci

Signed-off-by: limez <[email protected]>
@iimez
update examples / naming
b5e12e7 
Signed-off-by: limez <[email protected]>
@iimez
update deps, remove the need for binding.ci.gyp, make node-gyp-build …
1c40d3a 
…fallback easier testable

Signed-off-by: limez <[email protected]>
@iimez
make sure the assert-backend-sources.js script is published, but not …
f621d62 
…the others

Signed-off-by: limez <[email protected]>
@jacoobes @iimez
build correctly on windows (regression on node-gyp-build)
80bb306 
Signed-off-by: Jacob Nguyen <[email protected]>
@iimez
codespell
fd5705d 
Signed-off-by: limez <[email protected]>
@iimez
make sure dlhandle.cpp gets linked correctly
eaf4174 
Signed-off-by: limez <[email protected]>
@jacoobes jacoobes changed the title typescript bindings maintenance typescript bindings updates, cuda support, arm support, Aug 24, 2024
@jacoobes jacoobes changed the title typescript bindings updates, cuda support, arm support, typescript bindings updates, cuda support, linux arm64 support, Aug 24, 2024
@jacoobes jacoobes marked this pull request as draft August 24, 2024 03:32
@cebtenzzre cebtenzzre removed their request for review August 27, 2024 22:49
iimez and others added 8 commits September 2, 2024 02:00
@iimez
remove hasGpuDevice pr nomic-ai#2409
3b40a6f 
Signed-off-by: limez <[email protected]>
@iimez
ci
01aa868 
Signed-off-by: limez <[email protected]>
@jacoobes @iimez
lingering hasgpu
9a55815 
Signed-off-by: Jacob Nguyen <[email protected]>
@jacoobes @iimez
fix mac compilation?
7ce10bb 
Signed-off-by: Jacob Nguyen <[email protected]>
@jacoobes @iimez
fix linger
bfe34ac 
Signed-off-by: Jacob Nguyen <[email protected]>
@iimez
add getBackendName and getGpuDeviceName
650a97c 
Signed-off-by: limez <[email protected]>
@jacoobes @iimez
fix ci
edbd894 
Signed-off-by: Jacob Nguyen <[email protected]>
@iimez @jacoobes
merge win+mac ci fixes for node bindings from pull-2403
5fca68a 
Co-authored-by: Jacob Nguyen <[email protected]>
Signed-off-by: limez <[email protected]>
@iimez iimez changed the title typescript bindings updates, cuda support, linux arm64 support, typescript bindings updates, cuda support Sep 5, 2024
@jacoobes jacoobes marked this pull request as ready for review September 8, 2024 19:21
cebtenzzre
cebtenzzre reviewed Sep 9, 2024
View reviewed changes
.circleci/continue_config.yml
Comment on lines 1103 to +1105
build-bindings-backend-windows:
machine:
image: 'windows-server-2022-gui:2023.03.1'
image: 'windows-server-2019-vs2019:2022.08.1'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency with the other builds on main, please use VS2022, Enter-VsDevShell, Ninja, and ccache. Fix the merge conflicts first.

gpt4all-bindings/typescript/binding.gyp
@@ -27,7 +23,8 @@
'NAPI_CPP_EXCEPTIONS',
],
'cflags_cc': [
"-fexceptions"
"-fexceptions",
"-std=c++20"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The backend on main is configured in CMake to use C++23.

gpt4all-bindings/typescript/spec/chat-minimal.mjs
await createCompletion(chat, "Are you sure?", {
verbose: true,
});
);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a big deal, but I'd appreciate if you committed all text files with the trailing newline as per POSIX.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cebtenzzre cebtenzzre cebtenzzre left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@iimez @cebtenzzre @jacoobes