- passing include directories to `CUDAExtension`'s `include_dirs` argument
- removing `-I/path/to/dir` arguments from `extra_compile_args`

* support for fused dense layer with cublasLt, fusion in both fprop and bprop

* fix typo causing syntax error

* add fused GEMM+gelu+GEMM modue

* fix typo for workspace size

* update cublas check for 11600

* add tests for fused dense layer

* fix CUDA 10.x path

* safer guard around CUBLAS constants, remove unreferenced variable

* more guard changes

* guard against cublas version instead of cuda

Co-authored-by: Sukru Eryilmaz <seryilmaz@computelab-dgx1v-32.nvidia.com>

Optional NCCL communicator argument to init method

Bug fix in wgrad

Bug fixes

Various bug fixes in fused spatial parallel bottleneck block

* option to set param views to flat buffer

* remove redundant variables in init_stage1

Co-authored-by: Sukru Eryilmaz <seryilmaz@computelab-dgx1v-32.nvidia.com>
Co-authored-by: ptrblck <ptrblck@users.noreply.github.com>

* add full all-reduce code path

* debug

* debug

Co-authored-by: ptrblck <ptrblck@users.noreply.github.com>

Add functions to compute grad_out1, grad_out1_halo

* fuse norm into scale

* add fused norm into dlamb

Co-authored-by: Sukru Eryilmaz <seryilmaz@computelab-dgx1v-32.nvidia.com>

* support for fused dense layer with cublasLt, fusion in both fprop and bprop

* fix typo causing syntax error

* add fused GEMM+gelu+GEMM modue

* fix typo for workspace size

* update cublas check for 11600

* add tests for fused dense layer

* fix CUDA 10.x path

Co-authored-by: Sukru Eryilmaz <seryilmaz@computelab-dgx1v-32.nvidia.com>

wrapper function for flat view creation in _lazy_init_stage2

Spatially Distributed Fast Bottleneck block

Wrote a small wrapper function for flat view creation in _lazy_init_stage2 to support channels last data formats

* Added support for fused ReLU and dropout into transducer joint

* Reorganized code selection path in transducer joint fwd
* Added support for fused ReLU+dropout into transducer joint

* Vectorize transducer loss backward with fused softmax (#3)

* Nanz/transducer loss (#4)

* Vectorize transducer loss backward with fused softmax

* Added a predicate to avoid potential IMA

* Nanz/transducer loss (#5)

* Vectorize transducer loss backward with fused softmax

* Added a predicate to avoid potentional IMA

* Added more predicates to avoid IMAs

* Updated documentations for newly added features.

* Fixed a error in transducer.py

* local_rank and install cuda version fix

update fusedadam docstring

* clip before reduce scatter

* provide clip before/after RS option

* change to clip after ar (avoid confusion)

* fix comments

Co-authored-by: Sukru Eryilmaz <seryilmaz@computelab-dgx1v-32.nvidia.com>

* don't create cublasLt handle, fix zero block size case

* cleanup

* initial cublaslt support

* 64 bit input

* add license headers

* cleanup

* remove license

Co-authored-by: pbialecki <pbialecki@nvidia.com>

* initial commit for adding fast bottleneck

* sync cudnn-frontend module

Co-authored-by: pbialecki <pbialecki@nvidia.com>

Fixed a typo

* enable no_copy

* barrier for SHARP

* set verbose=False by default

Co-authored-by: Kexin Yu <kexiny@nvidia.com>

* Add unit tests for fused-novograd

* Fix: tensors should reside on the same device

* Fix: Cudastream should be called on the same device on which the tensors reside on. Found this during debugging fused novograd multi-device unit test

* fixed issues mentioned in the comments