twitter / the-algorithm-ml Goto Github PK

I think in each case the second definition overwrites the initial definition??? at least that is what I would expect.

projects/home/recap/models/config.py

BatchNormConfig is defined twice
DropoutConfig is defined twice
MlpConfig is defined twice

projects/home/recap/embedding/config.py

SmallEmbeddingBagConfig is defined twice

projects/home/recap/model/entrypoint.py

We have two imports stomping eachother out:

from tml.projects.home.recap.model import config as model_config_mod
import tml.projects.home.recap.model.config as model_config_mod

I can't find the script ./scripts/build_images.sh in twhin. Did you forget to commit it?"

I am really annoyed with people taking advantage of trending topics to post absolutely non related content for fun or marketing their services

Well, if asked what i think the solution would be, I would say an algorithm that checks tweets/posts for multiple unrelated hashtags and then shadowbans them would be a great start

please fix the timeline

Thank you, Elon!

We need more explanations on the ML Algorithm.

When I run the ./images/init_venv.sh,
then run this command:
python3.10 -m torch.distributed.run --standalone --nnodes 1 --nproc_per_node 1 tml/projects/twhin/run.py --config_yaml_path="tml/projects/twhin/config/local.yaml" --save_dir="tml/model"

And It told this error:
Traceback (most recent call last):
File "/DATA/jupyter/personal/twitter/tml/projects/twhin/run.py", line 105, in
app.run(main)
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
File "/DATA/jupyter/personal/twitter/tml/projects/twhin/run.py", line 97, in main
run(
File "/DATA/jupyter/personal/twitter/tml/projects/twhin/run.py", line 70, in run
ctl.train(
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/tml/core/custom_training_loop.py", line 192, in train
outputs = train_step_fn()
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/tml/core/custom_training_loop.py", line 57, in step_fn
outputs = pipeline.progress(data_iterator)
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/tml/core/train_pipeline.py", line 582, in progress
(losses, output) = cast(Tuple[torch.Tensor, Out], self._model(batch_i))
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/tml/projects/twhin/models/models.py", line 147, in forward
outputs = self.model(batch)
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/torchrec/distributed/model_parallel.py", line 255, in forward
return self._dmp_wrapped_module(*args, **kwargs)
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1040, in forward
output = self._run_ddp_forward(*inputs, **kwargs)
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1000, in _run_ddp_forward
return module_to_run(*inputs[0], **kwargs[0])
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/tml/projects/twhin/models/models.py", line 39, in forward
outs = self.large_embeddings(batch.nodes)
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/tml/common/modules/embedding/embedding.py", line 55, in forward
pooled_embs = self.ebc(sparse_features)
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/torchrec/distributed/types.py", line 594, in forward
dist_input = self.input_dist(ctx, *input, **kwargs).wait().wait()
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/torchrec/distributed/embeddingbag.py", line 424, in input_dist
input_dist(
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/torchrec/distributed/sharding/rw_sharding.py", line 303, in forward
) = bucketize_kjt_before_all2all(
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/torchrec/distributed/embedding_sharding.py", line 169, in bucketize_kjt_before_all2all
) = torch.ops.fbgemm.block_bucketize_sparse_features(
File "/DATA/jupyter/personal/twitter/tml/tml_venv/lib/python3.10/site-packages/torch/_ops.py", line 442, in call
return self._op(*args, **kwargs or {})
NotImplementedError: Could not run 'fbgemm::block_bucketize_sparse_features' with arguments from the 'CUDA' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'fbgemm::block_bucketize_sparse_features' is only available for these backends: [CPU, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradMPS, AutogradXPU, AutogradHPU, AutogradLazy, Tracer, AutocastCPU, AutocastCUDA, FuncTorchBatched, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PythonDispatcher].

Here is my CUDA version:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.129.06 Driver Version: 470.129.06 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... On | 00000000:3B:00.0 Off | 0 |
| N/A 33C P0 41W / 300W | 0MiB / 32510MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

Here is my pip list:
Package Version

absl-py 1.4.0
aiofiles 22.1.0
aiohttp 3.8.3
aiosignal 1.3.1
appdirs 1.4.4
arrow 1.2.3
asttokens 2.2.1
astunparse 1.6.3
async-timeout 4.0.2
attrs 22.1.0
backcall 0.2.0
black 22.6.0
cachetools 5.3.0
cblack 22.6.0
certifi 2022.12.7
cfgv 3.3.1
charset-normalizer 2.1.1
click 8.1.3
cmake 3.25.0
Cython 0.29.32
decorator 5.1.1
distlib 0.3.6
distro 1.8.0
dm-tree 0.1.6
docker 6.0.1
docker-pycreds 0.4.0
docstring-parser 0.8.1
exceptiongroup 1.1.0
executing 1.2.0
fbgemm-gpu-cpu 0.3.2
filelock 3.8.2
fire 0.5.0
flatbuffers 1.12
frozenlist 1.3.3
fsspec 2022.11.0
gast 0.4.0
gcsfs 2022.11.0
gitdb 4.0.10
GitPython 3.1.31
google-api-core 2.8.2
google-auth 2.16.0
google-auth-oauthlib 0.4.6
google-cloud-core 2.3.2
google-cloud-storage 2.7.0
google-crc32c 1.5.0
google-pasta 0.2.0
google-resumable-media 2.4.1
googleapis-common-protos 1.56.4
grpcio 1.51.1
h5py 3.8.0
hypothesis 6.61.0
identify 2.5.17
idna 3.4
importlib-metadata 6.0.0
iniconfig 2.0.0
iopath 0.1.10
ipdb 0.13.11
ipython 8.10.0
jedi 0.18.2
Jinja2 3.1.2
keras 2.9.0
Keras-Preprocessing 1.1.2
libclang 15.0.6.1
libcst 0.4.9
Markdown 3.4.1
MarkupSafe 2.1.1
matplotlib-inline 0.1.6
moreorless 0.4.0
multidict 6.0.4
mypy 1.0.1
mypy-extensions 0.4.3
nest-asyncio 1.5.6
ninja 1.11.1
nodeenv 1.7.0
numpy 1.22.0
nvidia-cublas-cu11 11.10.3.66
nvidia-cuda-nvrtc-cu11 11.7.99
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cudnn-cu11 8.5.0.96
oauthlib 3.2.2
opt-einsum 3.3.0
packaging 22.0
pandas 1.5.3
parso 0.8.3
pathspec 0.11.0
pathtools 0.1.2
pexpect 4.8.0
pickleshare 0.7.5
pip 23.1
platformdirs 3.0.0
pluggy 1.0.0
portalocker 2.6.0
portpicker 1.5.2
pre-commit 3.0.4
prompt-toolkit 3.0.36
protobuf 3.20.2
psutil 5.9.4
ptyprocess 0.7.0
pure-eval 0.2.2
pyarrow 10.0.1
pyasn1 0.4.8
pyasn1-modules 0.2.8
pydantic 1.9.0
pyDeprecate 0.3.2
Pygments 2.14.0
pyparsing 3.0.9
pyre-extensions 0.0.27
pytest 7.2.1
pytest-mypy 0.10.3
python-dateutil 2.8.2
pytz 2022.6
PyYAML 6.0
requests 2.28.1
requests-oauthlib 1.3.1
rsa 4.9
scikit-build 0.16.3
sentry-sdk 1.16.0
setproctitle 1.3.2
setuptools 65.5.0
six 1.16.0
smmap 5.0.0
sortedcontainers 2.4.0
stack-data 0.6.2
stdlibs 2022.10.9
tabulate 0.9.0
tensorboard 2.9.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.1
tensorflow 2.9.3
tensorflow-estimator 2.9.0
tensorflow-io-gcs-filesystem 0.30.0
termcolor 2.2.0
toml 0.10.2
tomli 2.0.1
torch 1.13.1
torchmetrics 0.11.0
torchrec 0.3.2
torchsnapshot 0.1.0
torchx 0.3.0
tqdm 4.64.1
trailrunner 1.2.1
traitlets 5.9.0
typing_extensions 4.4.0
typing-inspect 0.8.0
urllib3 1.26.13
usort 1.0.5
virtualenv 20.19.0
wandb 0.13.11
wcwidth 0.2.6
websocket-client 1.4.2
Werkzeug 2.2.3
wrapt 1.14.1
yarl 1.8.2
zipp 3.12.1

Thank you very much for helping me about this configure problem~

Hello.I wanna run the TwHIN model. I got this error in
twhin/optimizer.py. There is no fused_optimizer in twhin model definition when run the bulid_optimizer function

Python is slow. Server costs could be decreased by rewriting in Rust, which is safe and performant.

… is currently ongoing as per @jjh42 in #46 (comment)

might as well have a meta issue for that, both to track AND to avoid further contextually pointless pull requests

The README for the heavy ranker says that one of the engagements being predicted is scored_tweets_model_weight_retweet: The probability the user will Retweet the Tweet. Could you clarify if quote tweets counted as retweets for this prediction? Thanks!

Is your feature request related to a problem? Please describe.
Hey so thanks for the algorithm, however in order to run this is it possible to get some dummy de-anonymized/garbage data with some base behaviour established? So lets say I make a change, i should be able to see if it isn't a breaking change

Describe the solution you'd like
These could be a few of the ways

• Use git LFS to add model checkpoints/dummy data CSVs?
• Data Generation Distributions i.e a model to generate baseline data for the model

Describe alternatives you've considered
Generating self dummy data is prohibitively expensive and impractical frankly. Twitter can maybe create data streaming API for testing ML algorithms but I don't know how the pricing for that/load balancing even would work

Additional context
None needed so far

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment
For web, specify your OS and browser version. For mobile, specify device, OS
and version. For libraries, what version of build tools are you using?

Additional context
Add any other context about the problem here.

The Twitter algorithm has become an integral part of modern communication, allowing users to quickly and easily share thoughts, opinions, and news with a global audience. As a result, incorporating the Twitter algorithm into the C++ standard library would provide significant benefits to both developers and end-users.

Firstly, by including the Twitter algorithm in the C++ standard library, developers would be able to take advantage of a powerful tool that is already widely used and understood. This would make it easier for developers to integrate Twitter functionality into their applications, ultimately saving them time and effort.

Furthermore, the Twitter algorithm is constantly evolving and improving, which means that any updates or improvements to the algorithm would automatically be included in the standard library. This would ensure that developers always have access to the latest and most effective Twitter functionality, without having to worry about keeping their own libraries up to date.

In addition, including the Twitter algorithm in the C++ standard library would make it more accessible to a wider audience of developers. This would encourage more people to experiment with the algorithm and develop new and innovative applications, ultimately leading to a more dynamic and diverse software ecosystem.

Overall, adding the Twitter algorithm to the C++ standard library would provide numerous benefits to developers and end-users alike. By making it easier to integrate Twitter functionality into applications, providing access to the latest updates and improvements, and encouraging innovation and experimentation, this move would be a valuable addition to the C++ ecosystem.

Functional = more gooder

Figured this should have a tracking issue. See

Spotted via #50 ( thanks, @guibaq )

dcn just doesn't exist, never defined, never imported...Either someone is using dirty python magic to impute variables directly during runtime (I hope not), or this else block was never reached because an MLP was always used so it's a bug (I really hope not).

Describe the solution you'd like
Add OpenAI GPT-4, Midjourney V5 and GEN-2 Integration to the training dataloader so we can generate training data on the fly during each epoch. This data would have user specific context, thus we can later tweak 256B parameters to tailor each tweet specifically for the user.

This gives us ability to control the tweet generation making Anime Cat Girl tweets a possibility. This would allow users to stay calm and non-toxic and well behaved while using twitter, cause they always feel supportive.

Describe alternatives you've considered
Alternatively we can hire discord moderators to write training data in Microsoft Excel using copilot, then convert it to CSV.

Additional context

• https://penntoday.upenn.edu/news/tweets-twitter-users-could-predict-loneliness

Hi,

I would like to have the ability to search for tweets that I have seen on Twitter. This feature would allow me to easily find and reference tweets that I have previously come across, but may not have engaged with or saved.

The search function could be located within the "Explore" or "Search" tab, with an added filter option to search for "tweets I have seen." This would save time and effort in scrolling through my timeline or attempting to remember specific keywords or hashtags from the tweet.

Regards,
H. Zahera

Twitter's source code uses a slightly unusual code style, 2 space indentiation while the Python language preference is 4.

It would be nice if a .editorconfig file was included in each repo stating to use 2 space indention for .py files as that would help IDE's and editors use the correct indentation for the project.

$ ./projects/home/recap/scripts/create_random_data.sh

This will create some random data (in $HOME/tmp/recap_local_random_data).

$ ./projects/home/recap/scripts/run_local.sh

replace with
$ ./projects/home/recap/script/create_random_data.sh

This will create some random data (in $HOME/tmp/recap_local_random_data).

$ ./projects/home/recap/script/run_local.sh

Hi there,

Thanks for the laughs. This has inspired me (and hopefully others) to think about the code more seriously as well as the company in general.

Minuscule nitpicks.

1. Bad initialization gain value for ReLU nets.

   The initial weights for this feedforward block

   the-algorithm-ml/projects/home/recap/model/mask_net.py

   Line 39 in 78c3235

   self._mask_layer.apply(_init_weights)

   are set using

   the-algorithm-ml/projects/home/recap/model/mask_net.py

   Line 10 in 78c3235

   torch.nn.init.xavier_uniform_(module.weight)

   This will initialize the block with uniform weights with a default gain of 1.0. The recommended gain value for ReLU nets is sqrt(2) though. I see the same issue in other files as well.

   the-algorithm-ml/projects/home/recap/model/mlp.py

   Line 42 in 78c3235

   self.layers.apply(_init_weights)

   In practice, this rarely matters and should not affect the trained weights or deployed model.

2. Currently using nn.ModuleList when application calls for nn.Sequential

   It seems that the masknet is composed of primitive blocks that are either called in parallel or in sequence. In the parallel mode, the use of nn.ModuleList to hold the blocks makes sense. However, in the sequential mode, it might be more efficient to organize the blocks using nn.Sequential instead. Currenty, both are done using nn.Modulelist. This doesn't affect the correctness of the code however, it should slightly reduce the runtime during sequential application. Currently, the model is running a pythonic for loop over the blocks in sequence here

   the-algorithm-ml/projects/home/recap/model/mask_net.py

   Line 94 in 78c3235

   for mask_layer in self._mask_blocks:

   . Replacing with a nn.Sequential block would do the inference natively in C which should be faster and more scalable.

Thanks for making the repo public btw!

I cannot restore training model from checkpoint. The error log:
File "/data/zmining/jupyter-notebook/antnh/embeddings/tml/common/checkpointing/snapshot.py", line 67, in restore
snapshot.restore(self.state) #check
File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torchsnapshot/snapshot.py", line 406, in restore
self._load_stateful(
File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torchsnapshot/snapshot.py", line 671, in _load_stateful
stateful.load_state_dict(state_dict)
File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torch/nn/modules/module.py", line 2027, in load_state_dict
load(self, state_dict)
File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torch/nn/modules/module.py", line 2015, in load
load(child, child_state_dict, child_prefix)
File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torch/nn/modules/module.py", line 2015, in load
load(child, child_state_dict, child_prefix)
File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torch/nn/modules/module.py", line 2015, in load
load(child, child_state_dict, child_prefix)
[Previous line repeated 2 more times]
File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torch/nn/modules/module.py", line 2009, in load
module._load_from_state_dict(
File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1909, in _load_from_state_dict
hook(state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs)
File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torch/nn/modules/module.py", line 69, in call
return self.hook(module, *args, **kwargs)
File "/home/zdeploy/anaconda3/envs/quanhm_torchrec/lib/python3.8/site-packages/torchrec/distributed/embeddingbag.py", line 439, in _pre_load_state_dict_hook
local_shards = state_dict[key].local_shards()
KeyError: 'model._dmp_wrapped_module.module.large_embeddings.ebc.embedding_bags.user.weight'

Thank you very much for helping me.

Currently, there is no contributing.md file so there is nowhere to look to find direction on how to contribute if that is even possible.

Guys please don't abuse this section, keep it for bugs and contributions

Add a contributing guideline to the Readme for people to start contributing to the project.
Also the Readme.md needs a fix because currently it looks weird.

Elon, python is the reason not enough people are liking your tweets and Cat Turd is mad at you. Rust is a very advanced language with features to mitigate these issues though. Rewrite should be doable in a few days assuming you make everyone sleep at the office this weekend

.

Would include fishnet🐠 convolution matrix
For filtering fishy comments from recommendation😘

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment
For web, specify your OS and browser version. For mobile, specify device, OS
and version. For libraries, what version of build tools are you using?

Additional context
Add any other context about the problem here.

Some scripts depend on internal Twitter libraries (e.g., here). Please provide instruction on how to install them.