- 14 Oct, 2021 2 commits
-
-
Burc Eryilmaz authored
change chunking scheme for full-allreduce case, add parameter order argument, both to enable contiguous chunking of allgather (#1190)
-
Nan Zheng authored
1. remove the weight broadcast in the constructor 2. disable unnecessary allreduces for clip-after-ar
-
- 13 Oct, 2021 1 commit
-
-
eqy authored
-
- 09 Oct, 2021 1 commit
-
-
Masaki Kozuki authored
* run backward * remove custom_fwd/custom_bwd
-
- 08 Oct, 2021 1 commit
-
-
eqy authored
-
- 07 Oct, 2021 1 commit
-
-
eqy authored
-
- 06 Oct, 2021 1 commit
-
-
Masaki Kozuki authored
* [ColumnParallelLinear] Test behavior in autocast * fix test * casts manually to autocast dtype
-
- 02 Oct, 2021 1 commit
-
-
Masaki Kozuki authored
Co-authored-by:
Piotr Bialecki <pbialecki@nvidia.com> Co-authored-by:
Eddie Yan <eddiey@nvidia.com> Co-authored-by:
Rishi Puri <riship@nvidia.com> Co-authored-by:
Sangkug Lym <slym@nvidia.com>
-
- 01 Oct, 2021 1 commit
-
-
X Wang authored
-
- 29 Sep, 2021 1 commit
-
-
X Wang authored
-
- 25 Sep, 2021 1 commit
-
-
romerojosh authored
-
- 24 Sep, 2021 1 commit
-
-
Masaki Kozuki authored
-
- 08 Sep, 2021 1 commit
-
-
Masaki Kozuki authored
- passing include directories to `CUDAExtension`'s `include_dirs` argument - removing `-I/path/to/dir` arguments from `extra_compile_args`
-
- 04 Sep, 2021 1 commit
-
-
Burc Eryilmaz authored
* support for fused dense layer with cublasLt, fusion in both fprop and bprop * fix typo causing syntax error * add fused GEMM+gelu+GEMM modue * fix typo for workspace size * update cublas check for 11600 * add tests for fused dense layer * fix CUDA 10.x path * safer guard around CUBLAS constants, remove unreferenced variable * more guard changes * guard against cublas version instead of cuda Co-authored-by:
Sukru Eryilmaz <seryilmaz@computelab-dgx1v-32.nvidia.com>
-
- 03 Sep, 2021 2 commits
-
-
Thor Johnsen authored
Optional NCCL communicator argument to init method
-
Thor Johnsen authored
-
- 02 Sep, 2021 13 commits
-
-
Thor Johnsen authored
Bug fix in wgrad
-
Thor Johnsen authored
-
Thor Johnsen authored
Bug fixes
-
Thor Johnsen authored
-
Thor Johnsen authored
-
Thor Johnsen authored
Various bug fixes in fused spatial parallel bottleneck block
-
Thor Johnsen authored
-
Thor Johnsen authored
-
Burc Eryilmaz authored
* option to set param views to flat buffer * remove redundant variables in init_stage1 Co-authored-by:
Sukru Eryilmaz <seryilmaz@computelab-dgx1v-32.nvidia.com> Co-authored-by:
ptrblck <ptrblck@users.noreply.github.com>
-
Burc Eryilmaz authored
-
Kexin Yu authored
* add full all-reduce code path * debug * debug Co-authored-by:
ptrblck <ptrblck@users.noreply.github.com>
-
Thor Johnsen authored
Add functions to compute grad_out1, grad_out1_halo
-
Thor Johnsen authored
-
- 01 Sep, 2021 3 commits
-
-
Burc Eryilmaz authored
* fuse norm into scale * add fused norm into dlamb Co-authored-by:
Sukru Eryilmaz <seryilmaz@computelab-dgx1v-32.nvidia.com>
-
Burc Eryilmaz authored
* support for fused dense layer with cublasLt, fusion in both fprop and bprop * fix typo causing syntax error * add fused GEMM+gelu+GEMM modue * fix typo for workspace size * update cublas check for 11600 * add tests for fused dense layer * fix CUDA 10.x path Co-authored-by:
Sukru Eryilmaz <seryilmaz@computelab-dgx1v-32.nvidia.com>
-
Kexin Yu authored
wrapper function for flat view creation in _lazy_init_stage2
-
- 31 Aug, 2021 3 commits
-
-
Thor Johnsen authored
Spatially Distributed Fast Bottleneck block
-
Thor Johnsen authored
-
Thor Johnsen authored
-
- 30 Aug, 2021 1 commit
-
-
Thorsten Kurth authored
Wrote a small wrapper function for flat view creation in _lazy_init_stage2 to support channels last data formats
-
- 21 Aug, 2021 1 commit
-
-
X Wang authored
-
- 17 Jul, 2021 3 commits
-
-
Nan Zheng authored
* Added support for fused ReLU and dropout into transducer joint * Reorganized code selection path in transducer joint fwd * Added support for fused ReLU+dropout into transducer joint * Vectorize transducer loss backward with fused softmax (#3) * Nanz/transducer loss (#4) * Vectorize transducer loss backward with fused softmax * Added a predicate to avoid potential IMA * Nanz/transducer loss (#5) * Vectorize transducer loss backward with fused softmax * Added a predicate to avoid potentional IMA * Added more predicates to avoid IMAs * Updated documentations for newly added features. * Fixed a error in transducer.py
-
yjk21 authored
-
X Wang authored
* local_rank and install cuda version fix
-