Implement MLOps using Pytorch and Tensorflow framework. Easy configurable(Hydra) ML pipeline for tasks like Image Classification and Object Detection using SOLID engineering principles. https://github.com/aisingapore/PeekingDuck
-
Research and design industrial defect detection POC for American multinational company using OpenVINO. Deploying to Nvidia Jetson Xavier NX Developer kit.
+
Research and design industrial defect detection POC for American multinational company using OpenVINO. Deploying to Nvidia Jetson Xavier NX Developer kit.
-
Mentor batch 12 and 13 on Computer Vision.
+
Mentor batch 12 and 13 on Computer Vision.
AI Apprentice
@@ -399,7 +399,6 @@
AI Singapore : 2022 - 2023
Deploy scalable distributed container platforms using Kubernetes in GCP.
-
Career transition : 2021 - 2022 : 1 yrs
@@ -419,13 +418,14 @@
Career transition : 2021 - 2022 : Frontend implemented using React and Zustand.
-
+
NCS : 2020 - 2021 : 1 yrs
Fiori developer
Implement SAP Edge applications using Agile methodology. Maintenance Dashboard for Singapore Air force F16/Apache Squadron on Edge devices to improve turnaround reliability.
+
BHP : 2019 - 2020 : 1 yr 2 mos
Developer
diff --git a/search/search_index.json b/search/search_index.json
index 18eff2a..ad18849 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome to 124c41 type: short summary \u2502 \u2502 \u2502 \u2514\u2500\u2af8 Summary in present tense. Not capitalized. \u2502 \u2514\u2500\u2af8 Commit Type: build|cicd|docs|feat|fix|node|refactor|test AGI Humanity achieve AGI. Basic computing skills lost. NSFW content filtered for everyone. 1 Human decide to learn computing skills. NSFW content accessed. Home Server Chat : Chat with AI, RAG chatbot. Bot : AI assistant using GROQ, llama3 70B, RAG and web search. Stock Analysis Assistant : AI assistant using GROQ and llama3. ~~ Redlib : Reddit libre.~~ (killed by bots) Blog Ghost : Ghost blog. Beyond All Information : analyse your Beyond All Reason games. CheatSheets : Collection of cheatsheets. Cookbook : Collection of tech recipes. ~~ Forum : Host your own forum.~~ (deprecated) Neural Network Playground : Understand neural network visually. Note : Notepad Online. Use cookie storage only. linx : Image pastebin. pastebin : text/file Pastebin. speedtest : Speedtest. Home server : Build for ARM64 platform using Docker swarm mode. Team Fight Tactics ML : Analyse the current meta. Team Fight Tactics Strategy Application http://tftchamp.duckdns.org:3000/ Datasets publish @ https://www.kaggle.com/datasets/teckmengwong/team-fight-tactics-matches furyhawk/tftchamp: teamfight-tactics Data Analysis (github.com) About this dataset Team Fight Tactics highest ELO challengers games scrape by https://github.com/furyhawk/tftchamp. Using https://developer.riotgames.com/ API. 8 players FFA in one game. Target Label : placement 1 is best. Lower is better. Top 4 placement is a Win. Alternative prediction is to group Top 4 placement as Binary Win, bottom 4 as Binary Lost. Only team traits and augments/items chosen included in datasets. Stats like game_length , players_eliminated are excluded. This is to prevent the model from learning obvious predictor. sudo ./scripts/run_pipeline.sh -nrci Web Scraping With Python Objective This tutorial aims to show how to use the Python programming language to web scrape a website. Specifically, we will use the requests and Beautiful Soup libraries to scrape and parse data from companiesmarketcap.com and retrieve the \u201c Largest Companies by Market Cap \u201d. Finance details are scrape and parse from finance.yahoo.com . We will learn how to scale the web scraping process by first retrieving the first company/row of the table, then all companies on the website\u2019s first page, and finally, all 6024 companies from multiple pages. Once the scraping process is complete, we will preprocess the dataset and transform it into a more readable format before using matplotlib to visualise the most important information.","title":"Home"},{"location":"#welcome-to-124c41","text":"type: short summary \u2502 \u2502 \u2502 \u2514\u2500\u2af8 Summary in present tense. Not capitalized. \u2502 \u2514\u2500\u2af8 Commit Type: build|cicd|docs|feat|fix|node|refactor|test","title":"Welcome to 124c41"},{"location":"#agi","text":"Humanity achieve AGI. Basic computing skills lost. NSFW content filtered for everyone. 1 Human decide to learn computing skills. NSFW content accessed.","title":"AGI"},{"location":"#home-server","text":"Chat : Chat with AI, RAG chatbot. Bot : AI assistant using GROQ, llama3 70B, RAG and web search. Stock Analysis Assistant : AI assistant using GROQ and llama3. ~~ Redlib : Reddit libre.~~ (killed by bots) Blog Ghost : Ghost blog. Beyond All Information : analyse your Beyond All Reason games. CheatSheets : Collection of cheatsheets. Cookbook : Collection of tech recipes. ~~ Forum : Host your own forum.~~ (deprecated) Neural Network Playground : Understand neural network visually. Note : Notepad Online. Use cookie storage only. linx : Image pastebin. pastebin : text/file Pastebin. speedtest : Speedtest. Home server : Build for ARM64 platform using Docker swarm mode. Team Fight Tactics ML : Analyse the current meta.","title":"Home Server"},{"location":"#team-fight-tactics-strategy-application","text":"http://tftchamp.duckdns.org:3000/","title":"Team Fight Tactics Strategy Application"},{"location":"#datasets","text":"publish @ https://www.kaggle.com/datasets/teckmengwong/team-fight-tactics-matches furyhawk/tftchamp: teamfight-tactics Data Analysis (github.com)","title":"Datasets"},{"location":"#about-this-dataset","text":"Team Fight Tactics highest ELO challengers games scrape by https://github.com/furyhawk/tftchamp. Using https://developer.riotgames.com/ API. 8 players FFA in one game. Target Label : placement 1 is best. Lower is better. Top 4 placement is a Win. Alternative prediction is to group Top 4 placement as Binary Win, bottom 4 as Binary Lost. Only team traits and augments/items chosen included in datasets. Stats like game_length , players_eliminated are excluded. This is to prevent the model from learning obvious predictor. sudo ./scripts/run_pipeline.sh -nrci","title":"About this dataset"},{"location":"#web-scraping-with-python","text":"","title":"Web Scraping With Python"},{"location":"#objective","text":"This tutorial aims to show how to use the Python programming language to web scrape a website. Specifically, we will use the requests and Beautiful Soup libraries to scrape and parse data from companiesmarketcap.com and retrieve the \u201c Largest Companies by Market Cap \u201d. Finance details are scrape and parse from finance.yahoo.com . We will learn how to scale the web scraping process by first retrieving the first company/row of the table, then all companies on the website\u2019s first page, and finally, all 6024 companies from multiple pages. Once the scraping process is complete, we will preprocess the dataset and transform it into a more readable format before using matplotlib to visualise the most important information.","title":"Objective"},{"location":"about/","text":"Cervix defendite in atque Et Caesar ignes aspera Polymestoris dilectos obvius Lorem markdownum veli fratri tum illum coeptis, plagae? In armiferae rogum. Quas orsa Aeacide digitis huic solent moenia properata saetis, exitus dare gerunt viget ambage hac tamen terrae longe. Aura alvum, e requiescere, inrita ille foret cedere ego siquid deprensa culpetne ausa reparet, epota aequor ope . Ad sumptas medio , est cecidit more coniuge, viam. Est signaque parte, rogumque ensem, ubi bracchia Armeniae petis. Est nunc dicta terrae noluit. Nisi et caeco speciem expellam dolens praeceps Monstri collibus preces. Flavescunt tenuere Aegeus. click = alignment_point(rupSnapshot(vleOptic)) + uddiTooltipSnmp; if (leopardDriveE <= upTypeface(touchscreen - dvi, footer, cBootText.bar_heat.server(xhtmlWebcamAsp))) { compatibleMultithreadingTorrent.menu_pci(desktop_in, xmp_word, chipMashupDriver); browserUser = hypermedia_digitize_checksum(gigaflops_logic_winsock, cmos_pcb / clock_cron_drag, processor_cycle( keyboardOsJavascript, tweakCellListserv, control)); } type_kilohertz.standby_ram(microphone_null_osi, metaMedia); Adhibent cuius gentesque durasse dixit sterilique Meis me saxo; corpore pedibus. Undis Diomede! Se tellus ut sic illa et facti quamquam qualis fraudesque , conditus. Ulmo rurigenae proles prosiluit et plus movit. Titubantem inpono. Animalia pede, Minos nisi. Dixi stabat, curam curat vota sanguine Laestrygonis, ab materque finierat audes Thaumantias, terrae ulla, ego cetera. Serpentibus omnes exspectatum videre Aeneae thyrsos undae pietatis ulterius trementi agendum crudelis domitae. Fodiebant Caenis altae et Caeneus omnia ardor Consolor orbem tumentem, anguigenae sanguine tectis, lea glaebam guttae fuit valens caput, desubito te cursus aegre. Momordit misit solidumve Cereris cornu, illo dubiae: me est patris vias, mihi. Temptat spatiosi, cornu mater: cum Iris deus conchae tellusAndros tellus coniunx: sedes . Causa natorumque perque deponendique motu hoc facitote, lea quid quorum et multi venisse vox epulis nurusque? Duobus et corpora inerti, dea calamis equi quo iuncti thalamique starent suis est placido euntem ecce tertia. Nunc terrae flammas nec fuit minimamque, effugit ecce fulgentis in inter fertur: est sic colles ponderibus.","title":"About"},{"location":"about/#cervix-defendite-in-atque","text":"","title":"Cervix defendite in atque"},{"location":"about/#et-caesar-ignes-aspera-polymestoris-dilectos-obvius","text":"Lorem markdownum veli fratri tum illum coeptis, plagae? In armiferae rogum. Quas orsa Aeacide digitis huic solent moenia properata saetis, exitus dare gerunt viget ambage hac tamen terrae longe. Aura alvum, e requiescere, inrita ille foret cedere ego siquid deprensa culpetne ausa reparet, epota aequor ope . Ad sumptas medio , est cecidit more coniuge, viam. Est signaque parte, rogumque ensem, ubi bracchia Armeniae petis. Est nunc dicta terrae noluit.","title":"Et Caesar ignes aspera Polymestoris dilectos obvius"},{"location":"about/#nisi-et-caeco-speciem-expellam-dolens-praeceps","text":"Monstri collibus preces. Flavescunt tenuere Aegeus. click = alignment_point(rupSnapshot(vleOptic)) + uddiTooltipSnmp; if (leopardDriveE <= upTypeface(touchscreen - dvi, footer, cBootText.bar_heat.server(xhtmlWebcamAsp))) { compatibleMultithreadingTorrent.menu_pci(desktop_in, xmp_word, chipMashupDriver); browserUser = hypermedia_digitize_checksum(gigaflops_logic_winsock, cmos_pcb / clock_cron_drag, processor_cycle( keyboardOsJavascript, tweakCellListserv, control)); } type_kilohertz.standby_ram(microphone_null_osi, metaMedia);","title":"Nisi et caeco speciem expellam dolens praeceps"},{"location":"about/#adhibent-cuius-gentesque-durasse-dixit-sterilique","text":"Meis me saxo; corpore pedibus. Undis Diomede! Se tellus ut sic illa et facti quamquam qualis fraudesque , conditus. Ulmo rurigenae proles prosiluit et plus movit. Titubantem inpono. Animalia pede, Minos nisi. Dixi stabat, curam curat vota sanguine Laestrygonis, ab materque finierat audes Thaumantias, terrae ulla, ego cetera. Serpentibus omnes exspectatum videre Aeneae thyrsos undae pietatis ulterius trementi agendum crudelis domitae.","title":"Adhibent cuius gentesque durasse dixit sterilique"},{"location":"about/#fodiebant-caenis-altae-et-caeneus-omnia-ardor","text":"Consolor orbem tumentem, anguigenae sanguine tectis, lea glaebam guttae fuit valens caput, desubito te cursus aegre. Momordit misit solidumve Cereris cornu, illo dubiae: me est patris vias, mihi. Temptat spatiosi, cornu mater: cum Iris deus conchae tellusAndros tellus coniunx: sedes . Causa natorumque perque deponendique motu hoc facitote, lea quid quorum et multi venisse vox epulis nurusque? Duobus et corpora inerti, dea calamis equi quo iuncti thalamique starent suis est placido euntem ecce tertia. Nunc terrae flammas nec fuit minimamque, effugit ecce fulgentis in inter fertur: est sic colles ponderibus.","title":"Fodiebant Caenis altae et Caeneus omnia ardor"},{"location":"anomaly_detection/","text":"anomaly_detection pip install -e . pip install GitPython pip install onnx pip install openvino-dev Deep Learning for Anomaly Detection: A survey 1901.03407.pdf (arxiv.org) https://arxiv.org/pdf/1901.03407.pdf Type of Anomaly Point Anomalies represent an irregularity or deviation that happens randomly and may have no particular interpretation. Contextual Anomaly Detection conditional anomaly is a data instance that could be considered as anomalous in some specific context. Contextual anomaly is identified by considering both contextual and behavioural features. The contextual features, normally used are time and space. While the behavioral features may be a pattern of spending money, the occurrence of system log events or any feature used to describe the normal behavior. Collective or Group Anomaly Detection. Anomalous collections of individual data points are known as collective or group anomalies, wherein each of the individual points in isolation appears as normal data instances while observed in a group exhibit unusual characteristics. Output of DAD Techniques Anomaly Score Labels Diversity-Measurable Anomaly Detection | Papers With Code https://paperswithcode.com/paper/diversity-measurable-anomaly-detection Improvement of Autoencoders and Generative Adversarial Networks. Limitations . Focuses on anomaly with measurable geometrical diversity, the most common type in anomaly detection. However, as for anomaly with other kind of diversities, e.g. colors, the proposed diversity measure may not be positively correlated to anomaly severity. Attribute-based Representations for Accurate and Interpretable Video Anomaly Detection | Papers With Code https://paperswithcode.com/paper/attribute-based-representations-for-accurate Video anomaly detection Use object detection to extract features: Velocity Pose Deep features(pre-trained CLIP) Use kNN to detect anomaly on the extracted features. Could use similar approach to extract detective features. DSR -- A dual subspace re-projection network for surface anomaly detection | Papers With Code https://paperswithcode.com/paper/dsr-a-dual-subspace-re-projection-network-for Proposes an architecture based on quantized feature space representation with dual decoders, DSR, that avoids the image-level anomaly synthesis requirement. Without making any assumptions about the visual properties of anomalies, DSR generates the anomalies at the feature level by sampling the learned quantized feature space, which allows a controlled generation of near-in-distribution anomalies.","title":"anomaly_detection"},{"location":"anomaly_detection/#anomaly_detection","text":"pip install -e . pip install GitPython pip install onnx pip install openvino-dev","title":"anomaly_detection"},{"location":"anomaly_detection/#deep-learning-for-anomaly-detection-a-survey","text":"1901.03407.pdf (arxiv.org) https://arxiv.org/pdf/1901.03407.pdf","title":"Deep Learning for Anomaly Detection: A survey"},{"location":"anomaly_detection/#type-of-anomaly","text":"Point Anomalies represent an irregularity or deviation that happens randomly and may have no particular interpretation. Contextual Anomaly Detection conditional anomaly is a data instance that could be considered as anomalous in some specific context. Contextual anomaly is identified by considering both contextual and behavioural features. The contextual features, normally used are time and space. While the behavioral features may be a pattern of spending money, the occurrence of system log events or any feature used to describe the normal behavior. Collective or Group Anomaly Detection. Anomalous collections of individual data points are known as collective or group anomalies, wherein each of the individual points in isolation appears as normal data instances while observed in a group exhibit unusual characteristics. Output of DAD Techniques Anomaly Score Labels","title":"Type of Anomaly"},{"location":"anomaly_detection/#diversity-measurable-anomaly-detection-papers-with-code","text":"https://paperswithcode.com/paper/diversity-measurable-anomaly-detection Improvement of Autoencoders and Generative Adversarial Networks. Limitations . Focuses on anomaly with measurable geometrical diversity, the most common type in anomaly detection. However, as for anomaly with other kind of diversities, e.g. colors, the proposed diversity measure may not be positively correlated to anomaly severity.","title":"Diversity-Measurable Anomaly Detection | Papers With Code"},{"location":"anomaly_detection/#attribute-based-representations-for-accurate-and-interpretable-video-anomaly-detection-papers-with-code","text":"https://paperswithcode.com/paper/attribute-based-representations-for-accurate Video anomaly detection Use object detection to extract features: Velocity Pose Deep features(pre-trained CLIP) Use kNN to detect anomaly on the extracted features. Could use similar approach to extract detective features.","title":"Attribute-based Representations for Accurate and Interpretable Video Anomaly Detection | Papers With Code"},{"location":"anomaly_detection/#dsr-a-dual-subspace-re-projection-network-for-surface-anomaly-detection-papers-with-code","text":"https://paperswithcode.com/paper/dsr-a-dual-subspace-re-projection-network-for Proposes an architecture based on quantized feature space representation with dual decoders, DSR, that avoids the image-level anomaly synthesis requirement. Without making any assumptions about the visual properties of anomalies, DSR generates the anomalies at the feature level by sampling the learned quantized feature space, which allows a controlled generation of near-in-distribution anomalies.","title":"DSR -- A dual subspace re-projection network for surface anomaly detection | Papers With Code"},{"location":"ceph/","text":"Manual install of a Ceph Cluster. Daemon container This Dockerfile may be used to bootstrap a Ceph cluster with all the Ceph daemons running. To run a certain type of daemon, simply use the name of the daemon as $1 . Valid values are: mon deploys a Ceph monitor osd deploys an OSD using the method specified by OSD_TYPE osd_directory deploys one or multiple OSDs in a single container using a prepared directory (used in scenario where the operator doesn't want to use --privileged=true ) osd_directory_single deploys an single OSD per container using a prepared directory (used in scenario where the operator doesn't want to use --privileged=true ) osd_ceph_disk deploys an OSD using ceph-disk, so you have to provide a whole device (ie: /dev/sdb) mds deploys a MDS rgw deploys a Rados Gateway Usage You can use this container to bootstrap any Ceph daemon. CLUSTER is the name of the cluster (DEFAULT: ceph) SELinux If SELinux is enabled, run the following commands: sudo chcon -Rt svirt_sandbox_file_t /etc/ceph sudo chcon -Rt svirt_sandbox_file_t /var/lib/ceph KV backends We currently support one KV backend to store our configuration flags, keys and maps: etcd. There is a ceph.defaults config file in the image that is used for defaults to bootstrap daemons. It will add the keys if they are not already present. You can either pre-populate the KV store with your own settings, or provide a ceph.defaults config file. To supply your own defaults, make sure to mount the /etc/ceph/ volume and place your ceph.defaults file there. Important variables in ceph.defaults to add/change when you bootstrap an OSD: /osd/osd_journal_size /osd/cluster_network /osd/public_network Note: cluster_network and public_network are currently not populated in the defaults, but can be passed as environment variables with -e CEPH_PUBLIC_NETWORK=... for more flexibility Populate Key Value store docker run -d --net=host \\ -e KV_TYPE=etcd \\ -e KV_IP=127.0.0.1 \\ -e KV_PORT=2379 \\ ceph/daemon populate_kvstore Zap a device Sometimes you might want to destroy partition tables from a disk. For this you can use the zap_device scenario that works as follow: docker run -d --privileged=true \\ -v /dev/:/dev/ \\ -e OSD_DEVICE=/dev/sdd \\ ceph/daemon zap_device Deploy a monitor A monitor requires some persistent storage for the docker container. If a KV store is used, /etc/ceph will be auto-generated from data kept in the KV store. /var/lib/ceph , however, must be provided by a docker volume. The ceph mon will periodically store data into /var/lib/ceph , including the latest copy of the CRUSH map. If a mon restarts, it will attempt to download the latest monmap and CRUSH map from other peer monitors. However, if all mon daemons have gone down, monitors must be able to recover their previous maps. The docker volume used for /var/lib/ceph should be backed by some durable storage, and must be able to survive container and node restarts. Without KV store, run: docker run -d --net=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -e MON_IP=192.168.0.20 \\ -e CEPH_PUBLIC_NETWORK=192.168.0.0/24 \\ ceph/daemon mon With KV store, run: docker run -d --net=host \\ -v /var/lib/ceph:/var/lib/ceph \\ -e MON_IP=192.168.0.20 \\ -e CEPH_PUBLIC_NETWORK=192.168.0.0/24 \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon mon List of available options: MON_NAME : name of the monitor (default to hostname) CEPH_PUBLIC_NETWORK : CIDR of the host running Docker, it should be in the same network as the MON_IP CEPH_CLUSTER_NETWORK : CIDR of a secondary interface of the host running Docker. Used for the OSD replication traffic MON_IP : IP address of the host running Docker NETWORK_AUTO_DETECT : Whether and how to attempt IP and network autodetection. Meant to be used without --net=host . NEW_USER_KEYRING : if specified, it will be imported to keyrings. Works in demo mode only. 0 = Do not detect (default) 1 = Detect IPv6, fallback to IPv4 (if no globally-routable IPv6 address detected) 4 = Detect IPv4 only 6 = Detect IPv6 only Deploy a Manager daemon Since luminous, a manager daemon is mandatory, see docs Without KV store, run: docker run -d --net=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ ceph/daemon mgr With KV store, run: docker run -d --net=host \\ -v /var/lib/ceph:/var/lib/ceph \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon mgr Deploy an OSD There are four available OSD_TYPE values: - if no OSD_TYPE is set; one of disk , activate or directory will be used based on autodetection of the current OSD bootstrap state activate - the daemon expects to be passed a block device of a ceph-disk -prepared disk (via the OSD_DEVICE environment variable); no bootstrapping will be performed directory - the daemon expects to find the OSD filesystem(s) already mounted in /var/lib/ceph/osd/ disk - the daemon expects to be passed a block device via the OSD_DEVICE environment variable prepare - the daemon expects to be passed a block device and run ceph-disk prepare to bootstrap the disk (via the OSD_DEVICE environment variable) Options for OSDs (TODO: consolidate these options between the types): JOURNAL_DIR - if provided, new OSDs will be bootstrapped to use the specified directory as a common journal area. This is usually used to store the journals for more than one OSD on a common, separate disk. This currently only applies to the directory OSD type. JOURNAL - if provided, the new OSD will be bootstrapped to use the specified journal file (if you do not wish to use the default). This is currently only supported by the directory OSD type OSD_DEVICE - mandatory for activate and disk OSD types; this specifies which block device to use as the OSD OSD_JOURNAL - optional override of the OSD journal file. this only applies to the activate and disk OSD types OSD_FORCE_EXT4 - in case the osd data on ext4 is not automatically recognized (i.e. hidden by overlayfs) you can force them by settings this to yes . Without OSD_TYPE If the operator does not specify an OSD_TYPE autodetection happens: disk is used if no bootstrapped OSD is found. activate is used if a bootstrapped OSD is found and OSD_DEVICE is also provided. directory is used if a bootstrapped OSD is found and no OSD_DEVICE is provided. Without KV backend: docker run -d --net=host \\ --pid=host \\ --privileged=true \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ ceph/daemon osd With KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon osd Ceph disk Without KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ ceph/daemon osd Using bluestore: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e OSD_BLUESTORE=1 \\ ceph/daemon osd Using dmcrypt: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e OSD_DMCRYPT=1 \\ ceph/daemon osd With KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon osd Using bluestore with KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ -e OSD_BLUESTORE=1 \\ ceph/daemon osd Using dmcrypt with KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ -e OSD_DMCRYPT=1 \\ ceph/daemon osd List of available options: OSD_DEVICE is the OSD device OSD_JOURNAL is the journal for a given OSD HOSTNAME is used to place the OSD in the CRUSH map If you do not want to use --privileged=true , please fall back on the second example. Ceph disk activate This function is balance between ceph-disk and osd directory where the operator can use ceph-disk outside of the container (directly on the host) to prepare the devices. Devices will be prepared with ceph-disk prepare , then they will get activated inside the container. A priviledged container is still required as ceph-disk needs to access /dev/. So this has minimum value compare to the ceph-disk but might fit some use cases where the operators want to prepare their devices outside of a container. docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=activate \\ ceph/daemon osd Ceph OSD directory There are a number of environment variables which are used to configure the execution of the OSD: CLUSTER is the name of the ceph cluster (defaults to ceph ) If the OSD is not already created (key, configuration, OSD data), the following environment variables will control its creation: WEIGHT is the of the OSD when it is added to the CRUSH map (default is 1.0 ) JOURNAL is the location of the journal (default is the journal file inside the OSD data directory) HOSTNAME is the name of the host; it is used as a flag when adding the OSD to the CRUSH map The old option OSD_ID is now unused. Instead, the script will scan for each directory in /var/lib/ceph/osd of the form - . To create your OSDs simply run the following command: docker exec ceph osd create . Note that we now default to dropping root privileges, so it is important to set the proper ownership for your OSD directories. The Ceph OSD runs as UID:167, GID:167, so: chown -R 167:167 /var/lib/ceph/osd/ Multiple OSDs There is a problem when attempting run run multiple OSD containers on a single docker host. See issue #19. There are two workarounds, at present: Run each OSD with the --pid=host option Run multiple OSDs within the same container To run multiple OSDs within the same container, simply bind-mount each OSD datastore directory: docker run -v /osds/1:/var/lib/ceph/osd/ceph-1 -v /osds/2:/var/lib/ceph/osd/ceph-2 Ceph OSD directory single Ceph OSD directory single has a similar design to Ceph OSD directory since they both aim to run OSD processes from an already bootstrapped directory. So we assume the OSD directory has been populated already. The major different is that Ceph OSD directory single has a much simpler implementation since it only runs a single OSD process per container. It doesn't do anything with the journal as it assumes journal's symlink was provided during the initialization sequence of the OSD. This scenario goes through the OSD directory ( /var/lib/ceph/osd ) and looks for OSDs that don't have a lock held by any other OSD. If no lock is found, the OSD process starts. If all the OSDs are already running, we gently exit 0 and explain that all the OSDs are already running. Important note : if you are aiming at running multiple OSD containers on a same machine (things that you will likely do with Ceph anyway), you must enable --pid=host . However if you are running Docker 1.12 (based on https://github.com/docker/docker/pull/22481 ), you can just share the same PID namespace for the OSD containers only using: --pid=container: . BTRFS and journal If your OSD is BTRFS and you want to use PARALLEL journal mode, you will need to run this container with --privileged set to true. Otherwise, ceph-osd will have insufficient permissions and it will revert to the slower WRITEAHEAD mode. Note Re: [ https://github.com/Ulexus/docker-ceph/issues/5 ] A user has reported a consterning (and difficult to diagnose) problem wherein the OSD crashes frequently due to Docker running out of sufficient open file handles. This is understandable, as the OSDs use a great many ports during periods of high traffic. It is, therefore, recommended that you increase the number of open file handles available to Docker. On CoreOS (and probably other systemd-based systems), you can do this by creating the a file named /etc/systemd/system/docker.service.d/limits.conf with content something like: [Service] LimitNOFILE=4096 Deploy a MDS By default, the MDS does NOT create a ceph filesystem. If you wish to have this MDS create a ceph filesystem (it will only do this if the specified CEPHFS_NAME does not already exist), you must set, at a minimum, CEPHFS_CREATE=1 . It is strongly recommended that you read the rest of this section, as well. For most people, the defaults for the following optional environment variables are fine, but if you wish to customize the data and metadata pools in which your CephFS is stored, you may override the following as you wish: CEPHFS_CREATE : Whether to create the ceph filesystem (0 = no / 1 = yes), if it doesn't exist. Defaults to 0 (no) CEPHFS_NAME : The name of the new ceph filesystem and the basis on which the later variables are created. Defaults to cephfs CEPHFS_DATA_POOL : The name of the data pool for the ceph filesystem. If it does not exist, it will be created. Defaults to ${CEPHFS_NAME}_data CEPHFS_DATA_POOL_PG : The number of placement groups for the data pool. Defaults to 8 CEPHFS_METADATA_POOL : The name of the metadata pool for the ceph filesystem. If it does not exist, it will be created. Defaults to ${CEPHFS_NAME}_metadata CEPHFS_METADATA_POOL_PG : The number of placement groups for the metadata pool. Defaults to 8 Without KV backend, run: docker run -d --net=host \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /etc/ceph:/etc/ceph \\ -e CEPHFS_CREATE=1 \\ ceph/daemon mds With KV backend, run: docker run -d --net=host \\ -e CEPHFS_CREATE=1 \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon mds List of available options: MDS_NAME is the name the MDS server (DEFAULT: mds-$(hostname)). One thing to note is that metadata servers are not machine-restricted. They are not bound by their data directories and can move around the cluster. As a result, you can run more than one MDS on a single machine. If you plan to do so, you better set this variable and do something like: mds-$(hostname)-a , mds-$(hostname)-b etc... Deploy a Rados Gateway For the Rados Gateway, we deploy it with civetweb enabled by default. However it is possible to use different CGI frontends by simply giving remote address and port. Without kv backend, run: docker run -d --net=host \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /etc/ceph:/etc/ceph \\ ceph/daemon rgw With kv backend, run: docker run -d --net=host \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon rgw List of available options: RGW_CIVETWEB_PORT is the port to which civetweb is listening on (DEFAULT: 8080) RGW_NAME : default to hostname Administration via radosgw-admin from the Docker host if the RGW_NAME variable hasn't been supplied: docker exec radosgw-admin -n client.rgw.$(hostname) -k /var/lib/ceph/radosgw/$(hostname)/keyring If otherwise, $(hostname) has to be replaced by the value of RGW_NAME . To enable an external CGI interface instead of civetweb set: RGW_REMOTE_CGI=1 RGW_REMOTE_CGI_HOST=192.168.0.1 RGW_REMOTE_CGI_PORT=9000 And run the container like this docker run -d -v /etc/ceph:/etc/ceph -v /var/lib/ceph/:/var/lib/ceph -e CEPH_DAEMON=RGW -e RGW_NAME=myrgw -p 9000:9000 -e RGW_REMOTE_CGI=1 -e RGW_REMOTE_CGI_HOST=192.168.0.1 -e RGW_REMOTE_CGI_PORT=9000 ceph/daemon Deploy a REST API This is pretty straightforward. The --net=host is not mandatory, if you don't use it do not forget to expose the RESTAPI_PORT . Only available in luminous. docker run -d --net=host \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon restapi List of available options: RESTAPI_IP is the IP address to listen on (DEFAULT: 0.0.0.0) RESTAPI_PORT is the listening port of the REST API (DEFAULT: 5000) RESTAPI_BASE_URL is the base URL of the API (DEFAULT: /api/v0.1) RESTAPI_LOG_LEVEL is the log level of the API (DEFAULT: warning) RESTAPI_LOG_FILE is the location of the log file (DEFAULT: /var/log/ceph/ceph-restapi.log) Deploy a RBD mirror This is pretty straightforward. The --net=host is not mandatory, with KV we do: docker run -d --net=host \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon rbd_mirror Without KV we do: docker run -d --net=host \\ ceph/daemon rbd_mirror Fetching software. First of I want to check that I have all the latest packages in my debian system. apt update apt upgrade Next we fetch the keys and ceph packages, in this case we download the pacific packages for buster. wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add - echo deb https://download.ceph.com/debian-pacific/ buster main | sudo tee /etc/apt/sources.list.d/ceph.list apt update apt install ceph ceph-common Last we need to download the smartmontools for our nodes. This is so we can monitor our hard drives for hardware issues. echo deb http://deb.debian.org/debian buster-backports main >> /etc/apt/sources.list apt update apt install smartmontools/buster-backports A reboot when you have installed packages is always a good thing and if you need to do some extra hardware changes this is a good place to do so. shutdown -r now Configure node 1 First we will create a ceph configuration file. sudo vi /etc/ceph/ceph.conf The most important things to specify is the id and ips of your cluster monitors. A unique cluster id that you will reuse for all your nodes. And lastly a public network range that you want your monitors to be available over. The cluster network is a good addition if you have the resources to route the recovery traffic on a backbone network. [global] fsid = {cluster uuid} mon initial members = {id1}, {id2}, {id2} mon host = {ip1}, {ip2}, {ip3} public network = {network range for your public network} cluster network = {network range for your cluster network} auth cluster required = cephx auth service required = cephx auth client required = cephx Next we create keys for admin, monitors and boostrapping our drives. These keys will then be merged with the monitor key so the initial setup will have the keys used for other operations. sudo ceph-authtool --create-keyring /tmp/monkey --gen-key -n mon. --cap mon 'allow *' sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' sudo ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' sudo ceph-authtool /tmp/monkey --import-keyring /etc/ceph/ceph.client.admin.keyring sudo ceph-authtool /tmp/monkey --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring Make the monitor key available to the ceph user so we don't get an permission error when we start our services. sudo chown ceph:ceph /tmp/monkey Next up we create a monitor map so the monitors will know of each other. The monitors keeps track on other resources but for high availability the monitors needs to know who is in charge. monmaptool --create --add {node1-id} {node1-ip} --fsid {cluster uuid} /tmp/monmap monmaptool --add {node2-id} {node2-ip} --fsid {cluster uuid} /tmp/monmap monmaptool --add {node3-id} {node3-ip} --fsid {cluster uuid} /tmp/monmap Starting a new monitor is as easy as creating a new directory, creating the filesystem for and starting the service. sudo -u ceph mkdir /var/lib/ceph/mon/ceph-{node1-id} sudo -u ceph ceph-mon --mkfs -i {node1-id} --monmap /tmp/monmap --keyring /tmp/monkey sudo systemctl start ceph-mon@{node1-id} Next up we need a manager so we could configure and monitor our cluster through a visual dashboard. First we create a new key, put that key in a newly created directory and start the service. Enabling a dashboard is as easy as running the command for enabling, creating / assigning a certificate and creating a new admin user. sudo ceph auth get-or-create mgr.{node1-id} mon 'allow profile mgr' osd 'allow *' mds 'allow *' sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-{node1-id} sudo -u ceph vi /var/lib/ceph/mgr/ceph-{node1-id}/keyring sudo systemctl start ceph-mgr@{node1-id} sudo ceph mgr module enable dashboard sudo ceph dashboard create-self-signed-cert sudo ceph dashboard ac-user-create admin -i passwd administrator Setting up more nodes. First of we need to copy over the configuration, monitor map and all the keys over to our new host. sudo scp {user}@{server}:/etc/ceph/ceph.conf /etc/ceph/ceph.conf sudo scp {user}@{server}:/etc/ceph/ceph.client.admin.keyring /etc/ceph/ceph.client.admin.keyring sudo scp {user}@{server}:/var/lib/ceph/bootstrap-osd/ceph.keyring /var/lib/ceph/bootstrap-osd/ceph.keyring sudo scp {user}@{server}:/tmp/monmap /tmp/monmap sudo scp {user}@{server}:/tmp/monkey /tmp/monkey Next up we setup the monitor node exactly as we did with the first node. sudo -u ceph mkdir /var/lib/ceph/mon/ceph-{node2-id} sudo -u ceph ceph-mon --mkfs -i {node2-id} --monmap /tmp/monmap --keyring /tmp/monkey sudo systemctl start ceph-mon@{node2-id} sudo ceph -s sudo ceph mon enable-msgr2 Then we setup the manager node exactly as we did with the first node. sudo ceph auth get-or-create mgr.{node2-id} mon 'allow profile mgr' osd 'allow *' mds 'allow *' sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-{node2-id} sudo -u ceph vi /var/lib/ceph/mgr/ceph-{node2-id}/keyring sudo systemctl start ceph-mgr@{node2-id} Adding storage When the cluster is up and running and all monitors are in qourum you could add storage services. This is easily done via the volume command. First prepare a disk so it will be known by the cluster and have the keys and configuration copied to the management directory. Next up you activate the service so your storage nodes will be ready to use. This will be done for all the harddrives you want to add to your network. sudo ceph-volume lvm prepare --data /dev/sdb sudo ceph-volume lvm activate {osd-number} {osd-uuid} Post configuration Last but not least you want to ensure that all the services starts after a reboot. In debian you do that by enabling the services. sudo systemctl enable ceph-mon@{node-id} sudo systemctl enable ceph-mgr@{node-id} sudo systemctl enable ceph-osd@{osd-number}","title":"ceph"},{"location":"ceph/#manual-install-of-a-ceph-cluster","text":"","title":"Manual install of a Ceph Cluster."},{"location":"ceph/#daemon-container","text":"This Dockerfile may be used to bootstrap a Ceph cluster with all the Ceph daemons running. To run a certain type of daemon, simply use the name of the daemon as $1 . Valid values are: mon deploys a Ceph monitor osd deploys an OSD using the method specified by OSD_TYPE osd_directory deploys one or multiple OSDs in a single container using a prepared directory (used in scenario where the operator doesn't want to use --privileged=true ) osd_directory_single deploys an single OSD per container using a prepared directory (used in scenario where the operator doesn't want to use --privileged=true ) osd_ceph_disk deploys an OSD using ceph-disk, so you have to provide a whole device (ie: /dev/sdb) mds deploys a MDS rgw deploys a Rados Gateway","title":"Daemon container"},{"location":"ceph/#usage","text":"You can use this container to bootstrap any Ceph daemon. CLUSTER is the name of the cluster (DEFAULT: ceph)","title":"Usage"},{"location":"ceph/#selinux","text":"If SELinux is enabled, run the following commands: sudo chcon -Rt svirt_sandbox_file_t /etc/ceph sudo chcon -Rt svirt_sandbox_file_t /var/lib/ceph","title":"SELinux"},{"location":"ceph/#kv-backends","text":"We currently support one KV backend to store our configuration flags, keys and maps: etcd. There is a ceph.defaults config file in the image that is used for defaults to bootstrap daemons. It will add the keys if they are not already present. You can either pre-populate the KV store with your own settings, or provide a ceph.defaults config file. To supply your own defaults, make sure to mount the /etc/ceph/ volume and place your ceph.defaults file there. Important variables in ceph.defaults to add/change when you bootstrap an OSD: /osd/osd_journal_size /osd/cluster_network /osd/public_network Note: cluster_network and public_network are currently not populated in the defaults, but can be passed as environment variables with -e CEPH_PUBLIC_NETWORK=... for more flexibility","title":"KV backends"},{"location":"ceph/#populate-key-value-store","text":"docker run -d --net=host \\ -e KV_TYPE=etcd \\ -e KV_IP=127.0.0.1 \\ -e KV_PORT=2379 \\ ceph/daemon populate_kvstore","title":"Populate Key Value store"},{"location":"ceph/#zap-a-device","text":"Sometimes you might want to destroy partition tables from a disk. For this you can use the zap_device scenario that works as follow: docker run -d --privileged=true \\ -v /dev/:/dev/ \\ -e OSD_DEVICE=/dev/sdd \\ ceph/daemon zap_device","title":"Zap a device"},{"location":"ceph/#deploy-a-monitor","text":"A monitor requires some persistent storage for the docker container. If a KV store is used, /etc/ceph will be auto-generated from data kept in the KV store. /var/lib/ceph , however, must be provided by a docker volume. The ceph mon will periodically store data into /var/lib/ceph , including the latest copy of the CRUSH map. If a mon restarts, it will attempt to download the latest monmap and CRUSH map from other peer monitors. However, if all mon daemons have gone down, monitors must be able to recover their previous maps. The docker volume used for /var/lib/ceph should be backed by some durable storage, and must be able to survive container and node restarts. Without KV store, run: docker run -d --net=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -e MON_IP=192.168.0.20 \\ -e CEPH_PUBLIC_NETWORK=192.168.0.0/24 \\ ceph/daemon mon With KV store, run: docker run -d --net=host \\ -v /var/lib/ceph:/var/lib/ceph \\ -e MON_IP=192.168.0.20 \\ -e CEPH_PUBLIC_NETWORK=192.168.0.0/24 \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon mon List of available options: MON_NAME : name of the monitor (default to hostname) CEPH_PUBLIC_NETWORK : CIDR of the host running Docker, it should be in the same network as the MON_IP CEPH_CLUSTER_NETWORK : CIDR of a secondary interface of the host running Docker. Used for the OSD replication traffic MON_IP : IP address of the host running Docker NETWORK_AUTO_DETECT : Whether and how to attempt IP and network autodetection. Meant to be used without --net=host . NEW_USER_KEYRING : if specified, it will be imported to keyrings. Works in demo mode only. 0 = Do not detect (default) 1 = Detect IPv6, fallback to IPv4 (if no globally-routable IPv6 address detected) 4 = Detect IPv4 only 6 = Detect IPv6 only","title":"Deploy a monitor"},{"location":"ceph/#deploy-a-manager-daemon","text":"Since luminous, a manager daemon is mandatory, see docs Without KV store, run: docker run -d --net=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ ceph/daemon mgr With KV store, run: docker run -d --net=host \\ -v /var/lib/ceph:/var/lib/ceph \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon mgr","title":"Deploy a Manager daemon"},{"location":"ceph/#deploy-an-osd","text":"There are four available OSD_TYPE values: - if no OSD_TYPE is set; one of disk , activate or directory will be used based on autodetection of the current OSD bootstrap state activate - the daemon expects to be passed a block device of a ceph-disk -prepared disk (via the OSD_DEVICE environment variable); no bootstrapping will be performed directory - the daemon expects to find the OSD filesystem(s) already mounted in /var/lib/ceph/osd/ disk - the daemon expects to be passed a block device via the OSD_DEVICE environment variable prepare - the daemon expects to be passed a block device and run ceph-disk prepare to bootstrap the disk (via the OSD_DEVICE environment variable) Options for OSDs (TODO: consolidate these options between the types): JOURNAL_DIR - if provided, new OSDs will be bootstrapped to use the specified directory as a common journal area. This is usually used to store the journals for more than one OSD on a common, separate disk. This currently only applies to the directory OSD type. JOURNAL - if provided, the new OSD will be bootstrapped to use the specified journal file (if you do not wish to use the default). This is currently only supported by the directory OSD type OSD_DEVICE - mandatory for activate and disk OSD types; this specifies which block device to use as the OSD OSD_JOURNAL - optional override of the OSD journal file. this only applies to the activate and disk OSD types OSD_FORCE_EXT4 - in case the osd data on ext4 is not automatically recognized (i.e. hidden by overlayfs) you can force them by settings this to yes .","title":"Deploy an OSD"},{"location":"ceph/#without-osd_type","text":"If the operator does not specify an OSD_TYPE autodetection happens: disk is used if no bootstrapped OSD is found. activate is used if a bootstrapped OSD is found and OSD_DEVICE is also provided. directory is used if a bootstrapped OSD is found and no OSD_DEVICE is provided. Without KV backend: docker run -d --net=host \\ --pid=host \\ --privileged=true \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ ceph/daemon osd With KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon osd","title":"Without OSD_TYPE"},{"location":"ceph/#ceph-disk","text":"Without KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ ceph/daemon osd Using bluestore: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e OSD_BLUESTORE=1 \\ ceph/daemon osd Using dmcrypt: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e OSD_DMCRYPT=1 \\ ceph/daemon osd With KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon osd Using bluestore with KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ -e OSD_BLUESTORE=1 \\ ceph/daemon osd Using dmcrypt with KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ -e OSD_DMCRYPT=1 \\ ceph/daemon osd List of available options: OSD_DEVICE is the OSD device OSD_JOURNAL is the journal for a given OSD HOSTNAME is used to place the OSD in the CRUSH map If you do not want to use --privileged=true , please fall back on the second example.","title":"Ceph disk"},{"location":"ceph/#ceph-disk-activate","text":"This function is balance between ceph-disk and osd directory where the operator can use ceph-disk outside of the container (directly on the host) to prepare the devices. Devices will be prepared with ceph-disk prepare , then they will get activated inside the container. A priviledged container is still required as ceph-disk needs to access /dev/. So this has minimum value compare to the ceph-disk but might fit some use cases where the operators want to prepare their devices outside of a container. docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=activate \\ ceph/daemon osd","title":"Ceph disk activate"},{"location":"ceph/#ceph-osd-directory","text":"There are a number of environment variables which are used to configure the execution of the OSD: CLUSTER is the name of the ceph cluster (defaults to ceph ) If the OSD is not already created (key, configuration, OSD data), the following environment variables will control its creation: WEIGHT is the of the OSD when it is added to the CRUSH map (default is 1.0 ) JOURNAL is the location of the journal (default is the journal file inside the OSD data directory) HOSTNAME is the name of the host; it is used as a flag when adding the OSD to the CRUSH map The old option OSD_ID is now unused. Instead, the script will scan for each directory in /var/lib/ceph/osd of the form - . To create your OSDs simply run the following command: docker exec ceph osd create . Note that we now default to dropping root privileges, so it is important to set the proper ownership for your OSD directories. The Ceph OSD runs as UID:167, GID:167, so: chown -R 167:167 /var/lib/ceph/osd/","title":"Ceph OSD directory"},{"location":"ceph/#multiple-osds","text":"There is a problem when attempting run run multiple OSD containers on a single docker host. See issue #19. There are two workarounds, at present: Run each OSD with the --pid=host option Run multiple OSDs within the same container To run multiple OSDs within the same container, simply bind-mount each OSD datastore directory: docker run -v /osds/1:/var/lib/ceph/osd/ceph-1 -v /osds/2:/var/lib/ceph/osd/ceph-2","title":"Multiple OSDs"},{"location":"ceph/#ceph-osd-directory-single","text":"Ceph OSD directory single has a similar design to Ceph OSD directory since they both aim to run OSD processes from an already bootstrapped directory. So we assume the OSD directory has been populated already. The major different is that Ceph OSD directory single has a much simpler implementation since it only runs a single OSD process per container. It doesn't do anything with the journal as it assumes journal's symlink was provided during the initialization sequence of the OSD. This scenario goes through the OSD directory ( /var/lib/ceph/osd ) and looks for OSDs that don't have a lock held by any other OSD. If no lock is found, the OSD process starts. If all the OSDs are already running, we gently exit 0 and explain that all the OSDs are already running. Important note : if you are aiming at running multiple OSD containers on a same machine (things that you will likely do with Ceph anyway), you must enable --pid=host . However if you are running Docker 1.12 (based on https://github.com/docker/docker/pull/22481 ), you can just share the same PID namespace for the OSD containers only using: --pid=container: .","title":"Ceph OSD directory single"},{"location":"ceph/#btrfs-and-journal","text":"If your OSD is BTRFS and you want to use PARALLEL journal mode, you will need to run this container with --privileged set to true. Otherwise, ceph-osd will have insufficient permissions and it will revert to the slower WRITEAHEAD mode.","title":"BTRFS and journal"},{"location":"ceph/#note","text":"Re: [ https://github.com/Ulexus/docker-ceph/issues/5 ] A user has reported a consterning (and difficult to diagnose) problem wherein the OSD crashes frequently due to Docker running out of sufficient open file handles. This is understandable, as the OSDs use a great many ports during periods of high traffic. It is, therefore, recommended that you increase the number of open file handles available to Docker. On CoreOS (and probably other systemd-based systems), you can do this by creating the a file named /etc/systemd/system/docker.service.d/limits.conf with content something like: [Service] LimitNOFILE=4096","title":"Note"},{"location":"ceph/#deploy-a-mds","text":"By default, the MDS does NOT create a ceph filesystem. If you wish to have this MDS create a ceph filesystem (it will only do this if the specified CEPHFS_NAME does not already exist), you must set, at a minimum, CEPHFS_CREATE=1 . It is strongly recommended that you read the rest of this section, as well. For most people, the defaults for the following optional environment variables are fine, but if you wish to customize the data and metadata pools in which your CephFS is stored, you may override the following as you wish: CEPHFS_CREATE : Whether to create the ceph filesystem (0 = no / 1 = yes), if it doesn't exist. Defaults to 0 (no) CEPHFS_NAME : The name of the new ceph filesystem and the basis on which the later variables are created. Defaults to cephfs CEPHFS_DATA_POOL : The name of the data pool for the ceph filesystem. If it does not exist, it will be created. Defaults to ${CEPHFS_NAME}_data CEPHFS_DATA_POOL_PG : The number of placement groups for the data pool. Defaults to 8 CEPHFS_METADATA_POOL : The name of the metadata pool for the ceph filesystem. If it does not exist, it will be created. Defaults to ${CEPHFS_NAME}_metadata CEPHFS_METADATA_POOL_PG : The number of placement groups for the metadata pool. Defaults to 8 Without KV backend, run: docker run -d --net=host \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /etc/ceph:/etc/ceph \\ -e CEPHFS_CREATE=1 \\ ceph/daemon mds With KV backend, run: docker run -d --net=host \\ -e CEPHFS_CREATE=1 \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon mds List of available options: MDS_NAME is the name the MDS server (DEFAULT: mds-$(hostname)). One thing to note is that metadata servers are not machine-restricted. They are not bound by their data directories and can move around the cluster. As a result, you can run more than one MDS on a single machine. If you plan to do so, you better set this variable and do something like: mds-$(hostname)-a , mds-$(hostname)-b etc...","title":"Deploy a MDS"},{"location":"ceph/#deploy-a-rados-gateway","text":"For the Rados Gateway, we deploy it with civetweb enabled by default. However it is possible to use different CGI frontends by simply giving remote address and port. Without kv backend, run: docker run -d --net=host \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /etc/ceph:/etc/ceph \\ ceph/daemon rgw With kv backend, run: docker run -d --net=host \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon rgw List of available options: RGW_CIVETWEB_PORT is the port to which civetweb is listening on (DEFAULT: 8080) RGW_NAME : default to hostname Administration via radosgw-admin from the Docker host if the RGW_NAME variable hasn't been supplied: docker exec radosgw-admin -n client.rgw.$(hostname) -k /var/lib/ceph/radosgw/$(hostname)/keyring If otherwise, $(hostname) has to be replaced by the value of RGW_NAME . To enable an external CGI interface instead of civetweb set: RGW_REMOTE_CGI=1 RGW_REMOTE_CGI_HOST=192.168.0.1 RGW_REMOTE_CGI_PORT=9000 And run the container like this docker run -d -v /etc/ceph:/etc/ceph -v /var/lib/ceph/:/var/lib/ceph -e CEPH_DAEMON=RGW -e RGW_NAME=myrgw -p 9000:9000 -e RGW_REMOTE_CGI=1 -e RGW_REMOTE_CGI_HOST=192.168.0.1 -e RGW_REMOTE_CGI_PORT=9000 ceph/daemon","title":"Deploy a Rados Gateway"},{"location":"ceph/#deploy-a-rest-api","text":"This is pretty straightforward. The --net=host is not mandatory, if you don't use it do not forget to expose the RESTAPI_PORT . Only available in luminous. docker run -d --net=host \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon restapi List of available options: RESTAPI_IP is the IP address to listen on (DEFAULT: 0.0.0.0) RESTAPI_PORT is the listening port of the REST API (DEFAULT: 5000) RESTAPI_BASE_URL is the base URL of the API (DEFAULT: /api/v0.1) RESTAPI_LOG_LEVEL is the log level of the API (DEFAULT: warning) RESTAPI_LOG_FILE is the location of the log file (DEFAULT: /var/log/ceph/ceph-restapi.log)","title":"Deploy a REST API"},{"location":"ceph/#deploy-a-rbd-mirror","text":"This is pretty straightforward. The --net=host is not mandatory, with KV we do: docker run -d --net=host \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon rbd_mirror Without KV we do: docker run -d --net=host \\ ceph/daemon rbd_mirror","title":"Deploy a RBD mirror"},{"location":"ceph/#fetching-software","text":"First of I want to check that I have all the latest packages in my debian system. apt update apt upgrade Next we fetch the keys and ceph packages, in this case we download the pacific packages for buster. wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add - echo deb https://download.ceph.com/debian-pacific/ buster main | sudo tee /etc/apt/sources.list.d/ceph.list apt update apt install ceph ceph-common Last we need to download the smartmontools for our nodes. This is so we can monitor our hard drives for hardware issues. echo deb http://deb.debian.org/debian buster-backports main >> /etc/apt/sources.list apt update apt install smartmontools/buster-backports A reboot when you have installed packages is always a good thing and if you need to do some extra hardware changes this is a good place to do so. shutdown -r now","title":"Fetching software."},{"location":"ceph/#configure-node-1","text":"First we will create a ceph configuration file. sudo vi /etc/ceph/ceph.conf The most important things to specify is the id and ips of your cluster monitors. A unique cluster id that you will reuse for all your nodes. And lastly a public network range that you want your monitors to be available over. The cluster network is a good addition if you have the resources to route the recovery traffic on a backbone network. [global] fsid = {cluster uuid} mon initial members = {id1}, {id2}, {id2} mon host = {ip1}, {ip2}, {ip3} public network = {network range for your public network} cluster network = {network range for your cluster network} auth cluster required = cephx auth service required = cephx auth client required = cephx Next we create keys for admin, monitors and boostrapping our drives. These keys will then be merged with the monitor key so the initial setup will have the keys used for other operations. sudo ceph-authtool --create-keyring /tmp/monkey --gen-key -n mon. --cap mon 'allow *' sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' sudo ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' sudo ceph-authtool /tmp/monkey --import-keyring /etc/ceph/ceph.client.admin.keyring sudo ceph-authtool /tmp/monkey --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring Make the monitor key available to the ceph user so we don't get an permission error when we start our services. sudo chown ceph:ceph /tmp/monkey Next up we create a monitor map so the monitors will know of each other. The monitors keeps track on other resources but for high availability the monitors needs to know who is in charge. monmaptool --create --add {node1-id} {node1-ip} --fsid {cluster uuid} /tmp/monmap monmaptool --add {node2-id} {node2-ip} --fsid {cluster uuid} /tmp/monmap monmaptool --add {node3-id} {node3-ip} --fsid {cluster uuid} /tmp/monmap Starting a new monitor is as easy as creating a new directory, creating the filesystem for and starting the service. sudo -u ceph mkdir /var/lib/ceph/mon/ceph-{node1-id} sudo -u ceph ceph-mon --mkfs -i {node1-id} --monmap /tmp/monmap --keyring /tmp/monkey sudo systemctl start ceph-mon@{node1-id} Next up we need a manager so we could configure and monitor our cluster through a visual dashboard. First we create a new key, put that key in a newly created directory and start the service. Enabling a dashboard is as easy as running the command for enabling, creating / assigning a certificate and creating a new admin user. sudo ceph auth get-or-create mgr.{node1-id} mon 'allow profile mgr' osd 'allow *' mds 'allow *' sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-{node1-id} sudo -u ceph vi /var/lib/ceph/mgr/ceph-{node1-id}/keyring sudo systemctl start ceph-mgr@{node1-id} sudo ceph mgr module enable dashboard sudo ceph dashboard create-self-signed-cert sudo ceph dashboard ac-user-create admin -i passwd administrator","title":"Configure node 1"},{"location":"ceph/#setting-up-more-nodes","text":"First of we need to copy over the configuration, monitor map and all the keys over to our new host. sudo scp {user}@{server}:/etc/ceph/ceph.conf /etc/ceph/ceph.conf sudo scp {user}@{server}:/etc/ceph/ceph.client.admin.keyring /etc/ceph/ceph.client.admin.keyring sudo scp {user}@{server}:/var/lib/ceph/bootstrap-osd/ceph.keyring /var/lib/ceph/bootstrap-osd/ceph.keyring sudo scp {user}@{server}:/tmp/monmap /tmp/monmap sudo scp {user}@{server}:/tmp/monkey /tmp/monkey Next up we setup the monitor node exactly as we did with the first node. sudo -u ceph mkdir /var/lib/ceph/mon/ceph-{node2-id} sudo -u ceph ceph-mon --mkfs -i {node2-id} --monmap /tmp/monmap --keyring /tmp/monkey sudo systemctl start ceph-mon@{node2-id} sudo ceph -s sudo ceph mon enable-msgr2 Then we setup the manager node exactly as we did with the first node. sudo ceph auth get-or-create mgr.{node2-id} mon 'allow profile mgr' osd 'allow *' mds 'allow *' sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-{node2-id} sudo -u ceph vi /var/lib/ceph/mgr/ceph-{node2-id}/keyring sudo systemctl start ceph-mgr@{node2-id}","title":"Setting up more nodes."},{"location":"ceph/#adding-storage","text":"When the cluster is up and running and all monitors are in qourum you could add storage services. This is easily done via the volume command. First prepare a disk so it will be known by the cluster and have the keys and configuration copied to the management directory. Next up you activate the service so your storage nodes will be ready to use. This will be done for all the harddrives you want to add to your network. sudo ceph-volume lvm prepare --data /dev/sdb sudo ceph-volume lvm activate {osd-number} {osd-uuid}","title":"Adding storage"},{"location":"ceph/#post-configuration","text":"Last but not least you want to ensure that all the services starts after a reboot. In debian you do that by enabling the services. sudo systemctl enable ceph-mon@{node-id} sudo systemctl enable ceph-mgr@{node-id} sudo systemctl enable ceph-osd@{osd-number}","title":"Post configuration"},{"location":"chart/","text":"chart","title":"Chart"},{"location":"chart/#chart","text":"","title":"chart"},{"location":"code/","text":"code install vscode git clone https://AUR.archlinux.org/visual-studio-code-bin.git cd visual-studio-code-bin makepkg -s sudo pacman -U *code-bin-*.pkg.tar.zst cd ../ && sudo rm -rfv visual-studio-code-bin/","title":"vscode"},{"location":"code/#code","text":"","title":"code"},{"location":"code/#install-vscode","text":"git clone https://AUR.archlinux.org/visual-studio-code-bin.git cd visual-studio-code-bin makepkg -s sudo pacman -U *code-bin-*.pkg.tar.zst cd ../ && sudo rm -rfv visual-studio-code-bin/","title":"install vscode"},{"location":"conda/","text":"conda cheat sheet conda update -n base conda conda remove --name myenv --all conda env remove -n tf2 conda init powershell conda env create -f environment.yml conda env export > environment.yml conda create --name myenv python=3.10 conda install --file requirements.txt conda install --file requirements.txt -c conda-forge conda create --name tft --clone 311 pip install -r /path/to/requirements.txt pip install --upgrade --force-reinstall -r requirements.txt python -m ipykernel install --user --name myenv --display-name \"Python (myenv)\" conda clean -a conda install -y -c apple tensorflow-deps pip install tensorflow-macos tensorflow-metal bash Miniconda3-py310_22.11.1-1-Linux-x86_64.sh conda list conda config --add channels conda-forge conda config --describe conda config --set auto_activate_base false conda update --all pip --disable-pip-version-check list --outdated --format=json | python -c \"import json, sys; print('\\n'.join([x['name'] for x in json.load(sys.stdin)]))\" | xargs -n1 pip install -U conda update conda conda install --file requirements.txt -c conda-forge pip install -r requirements.txt # archlinux echo \"[ -f /opt/miniconda3/etc/profile.d/conda.sh ] && source /opt/miniconda3/etc/profile.d/conda.sh\" >> ~/.zshrc # ubuntu mkdir -p ~/miniconda3 wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 rm -rf ~/miniconda3/miniconda.sh source ~/miniconda3/etc/profile.d/conda.sh conda init zsh","title":"Conda"},{"location":"conda/#conda-cheat-sheet","text":"conda update -n base conda conda remove --name myenv --all conda env remove -n tf2 conda init powershell conda env create -f environment.yml conda env export > environment.yml conda create --name myenv python=3.10 conda install --file requirements.txt conda install --file requirements.txt -c conda-forge conda create --name tft --clone 311 pip install -r /path/to/requirements.txt pip install --upgrade --force-reinstall -r requirements.txt python -m ipykernel install --user --name myenv --display-name \"Python (myenv)\" conda clean -a conda install -y -c apple tensorflow-deps pip install tensorflow-macos tensorflow-metal bash Miniconda3-py310_22.11.1-1-Linux-x86_64.sh conda list conda config --add channels conda-forge conda config --describe conda config --set auto_activate_base false conda update --all pip --disable-pip-version-check list --outdated --format=json | python -c \"import json, sys; print('\\n'.join([x['name'] for x in json.load(sys.stdin)]))\" | xargs -n1 pip install -U conda update conda conda install --file requirements.txt -c conda-forge pip install -r requirements.txt # archlinux echo \"[ -f /opt/miniconda3/etc/profile.d/conda.sh ] && source /opt/miniconda3/etc/profile.d/conda.sh\" >> ~/.zshrc # ubuntu mkdir -p ~/miniconda3 wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 rm -rf ~/miniconda3/miniconda.sh source ~/miniconda3/etc/profile.d/conda.sh conda init zsh","title":"conda cheat sheet"},{"location":"dataframe/","text":"Dataframe df[\"cancer\"].value_counts().plot(kind='barh') df[[\"patient_id\", \"image_id\"]].astype(str).apply(lambda x: 'data/rsna/'+'/'.join(x)+'.png', axis=1) list(mapping.keys())[list(mapping.values()).index(x)] `` import logging from configs import LOGGER_NAME logger = logging.getLogger(LOGGER_NAME) # pylint: disable=invalid-name logger.info Rows of a pandas dataframe df def __len__(self) -> int: \"\"\"Return the length of the dataset.\"\"\" return len(self.df.index) In [7]: timeit len(df.index) 1000000 loops, best of 3: 248 ns per loop In [8]: timeit len(df) 1000000 loops, best of 3: 573 ns per loop","title":"Dataframe"},{"location":"dataframe/#dataframe","text":"df[\"cancer\"].value_counts().plot(kind='barh') df[[\"patient_id\", \"image_id\"]].astype(str).apply(lambda x: 'data/rsna/'+'/'.join(x)+'.png', axis=1) list(mapping.keys())[list(mapping.values()).index(x)] `` import logging from configs import LOGGER_NAME logger = logging.getLogger(LOGGER_NAME) # pylint: disable=invalid-name logger.info","title":"Dataframe"},{"location":"dataframe/#rows-of-a-pandas-dataframe-df","text":"def __len__(self) -> int: \"\"\"Return the length of the dataset.\"\"\" return len(self.df.index) In [7]: timeit len(df.index) 1000000 loops, best of 3: 248 ns per loop In [8]: timeit len(df) 1000000 loops, best of 3: 573 ns per loop","title":"Rows of a pandas dataframe df"},{"location":"db/","text":"SQLAlchemy conda create --name sql python=3.11 pip install SQLAlchemy pip install mariadb pip install PyMySQL brew install mariadb-connector-c brew install mysql docker-compose up --build -d","title":"database"},{"location":"db/#sqlalchemy","text":"conda create --name sql python=3.11 pip install SQLAlchemy pip install mariadb pip install PyMySQL brew install mariadb-connector-c brew install mysql docker-compose up --build -d","title":"SQLAlchemy"},{"location":"docker/","text":"Docker sudo apt update && sudo apt upgrade -y sudo apt install git curl htop mc rsync zsh-autosuggestions sudo apt install zsh chsh -s $(which zsh) sh -c \"$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)\" rsync -avuPz ./.oh-my-zsh/custom/themes/custom.zsh-theme furyhawk@arm:/home/furyhawk/.oh-my-zsh/custom/themes/custom.zsh-theme git clone https://github.com/zsh-users/zsh-autosuggestions.git $ZSH_CUSTOM/plugins/zsh-autosuggestions git clone https://github.com/zdharma-continuum/fast-syntax-highlighting.git ${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/plugins/fast-syntax-highlighting ## Enable plugins by adding them to .zshrc. - Open .zshrc `nano ~/.zshrc` - ZSH_THEME=\"custom\" - Find the line which says `plugins=(git)`. - Replace that line with `plugins=(git zsh-autosuggestions zsh-syntax-highlighting fast-syntax-highlighting zsh-autocomplete)` touch .zprofile nano .zprofile # set PATH so it includes user's private bin if it exists if [ -d \"$HOME/bin\" ] ; then PATH=\"$HOME/bin:$PATH\" fi # set PATH so it includes user's private bin if it exists if [ -d \"$HOME/.local/bin\" ] ; then PATH=\"$HOME/.local/bin:$PATH\" fi export TZ=Asia/Singapore export NODE_ID=$(docker info -f '{{.Swarm.NodeID}}') export EMAIL=furyx@hotmail.com export DOMAIN=furyhawk.lol sudo nano /etc/fstab touch .credentials nano .credentials sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc sudo chmod a+r /etc/apt/keyrings/docker.asc echo \\ \"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \\ $(. /etc/os-release && echo \"$VERSION_CODENAME\") stable\" | \\ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin # sudo groupadd docker sudo usermod -aG docker $USER sudo systemctl enable docker.service sudo systemctl enable containerd.service systemctl list-units --type=service --state=active cd /etc/docker sudo touch daemon.json sudo nano daemon.json { \"log-driver\": \"json-file\", \"log-opts\": { \"max-size\": \"10m\", \"max-file\": \"3\" } } docker swarm join --token SWMTKN-1-xxx 192.168.50.114:2377 docker node update --availability drain node docker node update --availability Active node curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm chmod +x cephadm sudo ./cephadm add-repo --release octopus sudo ./cephadm install cephadm install ceph-common mkdir -p /etc/ceph sudo ./cephadm bootstrap --mon-ip 192.168.65.19 URL: https://debian.local:8443/ User: admin Password: 12345678 You can access the Ceph CLI with: sudo ./cephadm shell --fsid 8ad7deb6-265a-11ef-9e81-c2d0c41fc7e0 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring Please consider enabling telemetry to help improve Ceph: ceph telemetry on For more information see: https://docs.ceph.com/docs/master/mgr/telemetry/ alias ceph='./cephadm shell -- ceph' sudo ceph -v ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) sudo ceph orch host label add debian mon Added label mon to host debian sudo ceph orch apply mon debian Scheduled mon update... sudo ceph orch host ls ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-2 ssh-copy-id -f -i /etc/ceph/ceph.pub osd1 ssh-copy-id -f -i /etc/ceph/ceph.pub osd2 ssh-copy-id -f -i /etc/ceph/ceph.pub osd3 sudo ceph orch host add osd1 sudo ceph orch host add osd2 sudo ceph orch host add osd3 sudo ceph orch apply osd --all-available-devices sudo ceph status sudo find / -iname 'cephadm*' 2>/dev/null systemctl --user start docker-desktop sudo groupadd docker sudo usermod -aG docker $USER groups ${USER} curl -fsSL test.docker.com -o get-docker.sh && sh get-docker.sh sudo apt-get install libffi-dev libssl-dev sudo apt install python3-dev sudo apt-get install -y python3 python3-pip pip install \"cython<3.0.0\" wheel pip install \"pyyaml==5.4.1\" --no-build-isolation pip install docker-compose sudo systemctl enable docker sudo apt-get purge docker-ce sudo apt-get purge docker-ce-cli sudo rm -rf /var/lib/docker rootless sudo apt install -y dbus-user-session sudo apt install -y fuse-overlayfs sudo apt install -y fuse-overlayfs sudo systemctl disable --now docker.service docker.socket sudo rm -rf /var/lib/docker sudo apt-get purge docker-ce docker-ce-cli containerd.io sudo apt-get purge docker-ce docker-ce-cli containerd.io apt-get install -y uidmap curl -fsSL https://get.docker.com/rootless | sh nano .zshrc export PATH=/home/furyhawk/bin:$PATH export DOCKER_HOST=unix:///run/user/1000/docker.sock systemctl --user enable docker sudo loginctl enable-linger $(whoami) docker context use rootless sudo setcap cap_net_bind_service=ep $(which rootlesskit) systemctl --user restart docker #sudo apt-get install -y docker-ce-rootless-extras dockerd-rootless-setuptool.sh uninstall cd ~/bin rm -f containerd containerd-shim containerd-shim-runc-v2 ctr docker docker-init docker-proxy dockerd dockerd-rootless-setuptool.sh dockerd-rootless.sh rootlesskit rootlesskit-docker-proxy runc vpnkit netstat -ltup The Compose Specification {:.no_toc} ToC {:toc} Status of this document This document specifies the Compose file format used to define multi-containers applications. Distribution of this document is unlimited. Requirements and optional attributes The Compose specification includes properties designed to target a local OCI container runtime, exposing Linux kernel specific configuration options, but also some Windows container specific properties. It is also designed for cloud platform features related to resource placement on a cluster, replicated application distribution, and scalability. We acknowledge that no Compose implementation is expected to support all attributes, and that support for some properties is platform dependent and can only be confirmed at runtime. The definition of a versioned schema to control the supported properties in a Compose file, established by the docker-compose tool where the Compose file format was designed, doesn't offer any guarantee to the end-user that attributes will be actually implemented. The specification defines the expected configuration syntax and behavior. Unless noted, supporting any of these is optional. A Compose implementation to parse a Compose file using unsupported attributes should warn users. We recommend the following implementors to support those running modes: Default: warn the user about unsupported attributes, but ignore them Strict: warn the user about unsupported attributes and reject the Compose file Loose: ignore unsupported attributes AND unknown attributes (that were not defined by the spec by the time implementation was created) From this point onwards, references made to 'Compose' can be interpreted as 'a Compose implementation'. The Compose application model The Compose Specification lets you define a platform-agnostic container based application. Such an application is designed as a set of containers which have to both run together with adequate shared resources and communication channels. Computing components of an application are defined as services . A service is an abstract concept implemented on platforms by running the same container image, and configuration, one or more times. Services communicate with each other through networks . In the Compose Specification, a network is a platform capability abstraction to establish an IP route between containers within services connected together. Low-level, platform-specific networking options are grouped into the Network definition and may be partially implemented on some platforms. Services store and share persistent data into volumes . The Specification describes such a persistent data as a high-level filesystem mount with global options. Actual platform-specific implementation details are grouped into the volumes definition and may be partially implemented on some platforms. Some services require configuration data that is dependent on the runtime or platform. For this, the Specification defines a dedicated configs concept. From a service container point of view, configs are comparable to volumes, in that they are files mounted into the container. But the actual definition involves distinct platform resources and services, which are abstracted by this type. A secret is a specific flavor of configuration data for sensitive data that should not be exposed without security considerations. Secrets are made available to services as files mounted into their containers, but the platform-specific resources to provide sensitive data are specific enough to deserve a distinct concept and definition within the Compose specification. Note With volumes, configs and secrets you can have a simple declaration at the top-level and then add more platform-specific information at the service level. A project is an individual deployment of an application specification on a platform. A project's name, set with the top-level name attribute, is used to group resources together and isolate them from other applications or other installation of the same Compose specified application with distinct parameters. If you are creating resources on a platform, you must prefix resource names by project and set the label com.docker.compose.project . Compose offers a way for users to set a custom project name and override this name, so that the same compose.yaml file can be deployed twice on the same infrastructure, without changes, by just passing a distinct name. Project names must contain only lowercase letters, decimal digits, dashes, and underscores, and must begin with a lowercase letter or decimal digit. Illustrative example The following example illustrates the Compose Specification concepts outlined above. The example is non-normative. Consider an application split into a frontend web application and a backend service. The frontend is configured at runtime with an HTTP configuration file managed by infrastructure, providing an external domain name, and an HTTPS server certificate injected by the platform's secured secret store. The backend stores data in a persistent volume. Both services communicate with each other on an isolated back-tier network, while the frontend is also connected to a front-tier network and exposes port 443 for external usage. %%{ init: { 'flowchart': { 'curve': 'linear' } } }%% flowchart LR subgraph A[INFRASTRUCTURE] direction TB subgraph TOP[\" \"] subgraph B1[Frontend Service] fs[\"`**webapp**`\"] end style B1 fill:#ccd6e8, stroke-width:0px subgraph B2[Backend Service] bs[\"`**database**`\"] end style B2 fill:#ccd6e8, stroke-width:0px end style TOP fill:transparent, stroke-width:2px, stroke:#62affb, stroke-dasharray: 5 5 key[ro= read only\\nr+w = read write] style key fill:transparent, stroke-width:0px,text-align: left, size: 94px direction TB id2(Server\\nCertificate) id1(HTTP\\nConfiguration) id1 & id2 -.-|ro| B1 style id1 stroke:#000,stroke-width:1px,stroke-dasharray: 10 style id2 stroke:#000,stroke-width:1px,stroke-dasharray: 10 B2 ==r+w==> id3[(Persistent\\nVolume)] end style A fill:#eeeeee, stroke-width:0px direction LR id4[External\\nUser] ---id5(((443)))--->|Frontend\\nNetwork| B1 style id4 stroke:#000,stroke-width:2px B1 --Backend\\nNetwork--> B2 The example application is composed of the following parts: 2 services, backed by Docker images: webapp and database 1 secret (HTTPS certificate), injected into the frontend 1 configuration (HTTP), injected into the frontend 1 persistent volume, attached to the backend 2 networks services: frontend: image: example/webapp ports: - \"443:8043\" networks: - front-tier - back-tier configs: - httpd-config secrets: - server-certificate backend: image: example/database volumes: - db-data:/etc/data networks: - back-tier volumes: db-data: driver: flocker driver_opts: size: \"10GiB\" configs: httpd-config: external: true secrets: server-certificate: external: true networks: ## The presence of these objects is sufficient to define them front-tier: {} back-tier: {} This example illustrates the distinction between volumes, configs and secrets. While all of them are all exposed to service containers as mounted files or directories, only a volume can be configured for read+write access. Secrets and configs are read-only. The volume configuration allows you to select a volume driver and pass driver options to tweak volume management according to the actual infrastructure. Configs and secrets rely on platform services, and are declared external as they are not managed as part of the application lifecycle. Compose uses a platform-specific lookup mechanism to retrieve runtime values. Compose file The Compose file is a YAML file defining: - Version (Optional) - Services (Required) - Networks - Volumes - Configs - Secrets The default path for a Compose file is compose.yaml (preferred) or compose.yml that is placed in the working directory. Compose also supports docker-compose.yaml and docker-compose.yml for backwards compatibility of earlier versions. If both files exist, Compose prefers the canonical compose.yaml . You can use fragments and extensions to keep your Compose file efficient and easy to maintain. Multiple Compose files can be merged together to define the application model. The combination of YAML files are implemented by appending or overriding YAML elements based on the Compose file order you set. Simple attributes and maps get overridden by the highest order Compose file, lists get merged by appending. Relative paths are resolved based on the first Compose file's parent folder, whenever complimentary files being merged are hosted in other folders. As some Compose file elements can both be expressed as single strings or complex objects, merges apply to the expanded form. If you want to reuse other Compose files, or factor out parts of you application model into separate Compose files, you can also use include . This is useful if your Compose application is dependent on another application which is managed by a different team, or needs to be shared with others. Version and name top-level elements Version top-level element The top-level version property is defined by the Compose Specification for backward compatibility. It is only informative. Compose doesn't use version to select an exact schema to validate the Compose file, but prefers the most recent schema when it's implemented. Compose validates whether it can fully parse the Compose file. If some fields are unknown, typically because the Compose file was written with fields defined by a newer version of the Specification, you'll receive a warning message. Compose offers options to ignore unknown fields (as defined by \"loose\" mode). Name top-level element The top-level name property is defined by the Specification as the project name to be used if you don't set one explicitly. Compose offers a way for you to override this name, and sets a default project name to be used if the top-level name element is not set. Whenever a project name is defined by top-level name or by some custom mechanism, it is exposed for interpolation and environment variable resolution as COMPOSE_PROJECT_NAME services: foo: image: busybox environment: - COMPOSE_PROJECT_NAME command: echo \"I'm running ${COMPOSE_PROJECT_NAME}\" Services top-level element A service is an abstract definition of a computing resource within an application which can be scaled or replaced independently from other components. Services are backed by a set of containers, run by the platform according to replication requirements and placement constraints. As services are backed by containers, they are defined by a Docker image and set of runtime arguments. All containers within a service are identically created with these arguments. A Compose file must declare a services top-level element as a map whose keys are string representations of service names, and whose values are service definitions. A service definition contains the configuration that is applied to each service container. Each service may also include a build section, which defines how to create the Docker image for the service. Compose supports building docker images using this service definition. If not used, the build section is ignored and the Compose file is still considered valid. Build support is an optional aspect of the Compose Specification, and is described in detail in the Compose Build Specification documentation. Each service defines runtime constraints and requirements to run its containers. The deploy section groups these constraints and allows the platform to adjust the deployment strategy to best match containers' needs with available resources. Deploy support is an optional aspect of the Compose Specification, and is described in detail in the Compose Deploy Specification documentation. If not implemented the deploy section is ignored and the Compose file is still considered valid. attach When attach is defined and set to false Compose does not collect service logs, until you explicitly request it to. The default service configuration is attach: true . build build specifies the build configuration for creating a container image from source, as defined in the Compose Build Specification . blkio_config blkio_config defines a set of configuration options to set block IO limits for a service. services: foo: image: busybox blkio_config: weight: 300 weight_device: - path: /dev/sda weight: 400 device_read_bps: - path: /dev/sdb rate: '12mb' device_read_iops: - path: /dev/sdb rate: 120 device_write_bps: - path: /dev/sdb rate: '1024k' device_write_iops: - path: /dev/sdb rate: 30 device_read_bps, device_write_bps Set a limit in bytes per second for read / write operations on a given device. Each item in the list must have two keys: path : Defines the symbolic path to the affected device. rate : Either as an integer value representing the number of bytes or as a string expressing a byte value. device_read_iops, device_write_iops Set a limit in operations per second for read / write operations on a given device. Each item in the list must have two keys: path : Defines the symbolic path to the affected device. rate : As an integer value representing the permitted number of operations per second. weight Modify the proportion of bandwidth allocated to a service relative to other services. Takes an integer value between 10 and 1000, with 500 being the default. weight_device Fine-tune bandwidth allocation by device. Each item in the list must have two keys: path : Defines the symbolic path to the affected device. weight : An integer value between 10 and 1000. cpu_count cpu_count defines the number of usable CPUs for service container. cpu_percent cpu_percent defines the usable percentage of the available CPUs. cpu_shares cpu_shares defines, as integer value, a service container's relative CPU weight versus other containers. cpu_period cpu_period configures CPU CFS (Completely Fair Scheduler) period when a platform is based on Linux kernel. cpu_quota cpu_quota configures CPU CFS (Completely Fair Scheduler) quota when a platform is based on Linux kernel. cpu_rt_runtime cpu_rt_runtime configures CPU allocation parameters for platforms with support for realtime scheduler. It can be either an integer value using microseconds as unit or a duration . cpu_rt_runtime: '400ms' cpu_rt_runtime: 95000` cpu_rt_period cpu_rt_period configures CPU allocation parameters for platforms with support for realtime scheduler. It can be either an integer value using microseconds as unit or a duration . cpu_rt_period: '1400us' cpu_rt_period: 11000` cpus DEPRECATED: use deploy.limits.cpus cpus define the number of (potentially virtual) CPUs to allocate to service containers. This is a fractional number. 0.000 means no limit. cpuset cpuset defines the explicit CPUs in which to allow execution. Can be a range 0-3 or a list 0,1 cap_add cap_add specifies additional container capabilities as strings. cap_add: - ALL cap_drop cap_drop specifies container capabilities to drop as strings. cap_drop: - NET_ADMIN - SYS_ADMIN cgroup cgroup specifies the cgroup namespace to join. When unset, it is the container runtime's decision to select which cgroup namespace to use, if supported. host : Runs the container in the Container runtime cgroup namespace. private : Runs the container in its own private cgroup namespace. cgroup_parent cgroup_parent specifies an optional parent cgroup for the container. cgroup_parent: m-executor-abcd command command overrides the default command declared by the container image, for example by Dockerfile's CMD . command: bundle exec thin -p 3000 The value can also be a list, in a manner similar to Dockerfile : command: [ \"bundle\", \"exec\", \"thin\", \"-p\", \"3000\" ] If the value is null , the default command from the image is used. If the value is [] (empty list) or '' (empty string), the default command declared by the image is ignored, i.e. overridden to be empty. configs Configs allow services to adapt their behaviour without the need to rebuild a Docker image. Services can only access configs when explicitly granted by the configs attribute. Two different syntax variants are supported. Compose reports an error if config doesn't exist on the platform or isn't defined in the configs top-level element in the Compose file. There are two syntaxes defined for configs. To remain compliant to this specification, an implementation must support both syntaxes. Implementations must allow use of both short and long syntaxes within the same document. You can grant a service access to multiple configs, and you can mix long and short syntax. Short syntax The short syntax variant only specifies the config name. This grants the container access to the config and mounts it as files into a service\u2019s container\u2019s filesystem. The location of the mount point within the container defaults to / in Linux containers, and C:\\ in Windows containers. The following example uses the short syntax to grant the redis service access to the my_config and my_other_config configs. The value of my_config is set to the contents of the file ./my_config.txt , and my_other_config is defined as an external resource, which means that it has already been defined in the platform. If the external config does not exist, the deployment fails. services: redis: image: redis:latest configs: - my_config - my_other_config configs: my_config: file: ./my_config.txt my_other_config: external: true Long syntax The long syntax provides more granularity in how the config is created within the service's task containers. source : The name of the config as it exists in the platform. target : The path and name of the file to be mounted in the service's task containers. Defaults to / if not specified. uid and gid : The numeric UID or GID that owns the mounted config file within the service's task containers. Default value when not specified is USER running container. mode : The permissions for the file that is mounted within the service's task containers, in octal notation. Default value is world-readable ( 0444 ). Writable bit must be ignored. The executable bit can be set. The following example sets the name of my_config to redis_config within the container, sets the mode to 0440 (group-readable) and sets the user and group to 103 . The redis service does not have access to the my_other_config config. services: redis: image: redis:latest configs: - source: my_config target: /redis_config uid: \"103\" gid: \"103\" mode: 0440 configs: my_config: external: true my_other_config: external: true container_name container_name is a string that specifies a custom container name, rather than a name generated by default. container_name: my-web-container Compose does not scale a service beyond one container if the Compose file specifies a container_name . Attempting to do so results in an error. container_name follows the regex format of [a-zA-Z0-9][a-zA-Z0-9_.-]+ credential_spec credential_spec configures the credential spec for a managed service account. If you have services that use Windows containers, you can use file: and registry: protocols for credential_spec . Compose also supports additional protocols for custom use-cases. The credential_spec must be in the format file:// or registry:// . credential_spec: file: my-credential-spec.json When using registry: , the credential spec is read from the Windows registry on the daemon's host. A registry value with the given name must be located in: HKLM\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Virtualization\\Containers\\CredentialSpecs The following example loads the credential spec from a value named my-credential-spec in the registry: credential_spec: registry: my-credential-spec Example gMSA configuration When configuring a gMSA credential spec for a service, you only need to specify a credential spec with config , as shown in the following example: services: myservice: image: myimage:latest credential_spec: config: my_credential_spec configs: my_credentials_spec: file: ./my-credential-spec.json| depends_on depends_on expresses startup and shutdown dependencies between services. Short syntax The short syntax variant only specifies service names of the dependencies. Service dependencies cause the following behaviors: Compose creates services in dependency order. In the following example, db and redis are created before web . Compose removes services in dependency order. In the following example, web is removed before db and redis . Simple example: services: web: build: . depends_on: - db - redis redis: image: redis db: image: postgres Compose guarantees dependency services have been started before starting a dependent service. Compose waits for dependency services to be \"ready\" before starting a dependent service. Long syntax The long form syntax enables the configuration of additional fields that can't be expressed in the short form. restart : When set to true Compose restarts this service after it updates the dependency service. This applies to an explicit restart controlled by a Compose operation, and excludes automated restart by the container runtime after the container dies. condition : Sets the condition under which dependency is considered satisfied service_started : An equivalent of the short syntax described above service_healthy : Specifies that a dependency is expected to be \"healthy\" (as indicated by healthcheck ) before starting a dependent service. service_completed_successfully : Specifies that a dependency is expected to run to successful completion before starting a dependent service. required : When set to false Compose only warns you when the dependency service isn't started or available. If it's not defined the default value of required is true . Service dependencies cause the following behaviors: Compose creates services in dependency order. In the following example, db and redis are created before web . Compose waits for healthchecks to pass on dependencies marked with service_healthy . In the following example, db is expected to be \"healthy\" before web is created. Compose removes services in dependency order. In the following example, web is removed before db and redis . services: web: build: . depends_on: db: condition: service_healthy restart: true redis: condition: service_started redis: image: redis db: image: postgres Compose guarantees dependency services are started before starting a dependent service. Compose guarantees dependency services marked with service_healthy are \"healthy\" before starting a dependent service. deploy deploy specifies the configuration for the deployment and lifecycle of services, as defined in the Compose Deploy Specification . develop develop specifies the development configuration for maintaining a container in sync with source, as defined in the Development Section . device_cgroup_rules device_cgroup_rules defines a list of device cgroup rules for this container. The format is the same format the Linux kernel specifies in the Control Groups Device Whitelist Controller . device_cgroup_rules: - 'c 1:3 mr' - 'a 7:* rmw' devices devices defines a list of device mappings for created containers in the form of HOST_PATH:CONTAINER_PATH[:CGROUP_PERMISSIONS] . devices: - \"/dev/ttyUSB0:/dev/ttyUSB0\" - \"/dev/sda:/dev/xvda:rwm\" dns dns defines custom DNS servers to set on the container network interface configuration. It can be a single value or a list. dns: 8.8.8.8 dns: - 8.8.8.8 - 9.9.9.9 dns_opt dns_opt list custom DNS options to be passed to the container\u2019s DNS resolver ( /etc/resolv.conf file on Linux). dns_opt: - use-vc - no-tld-query dns_search dns_search defines custom DNS search domains to set on container network interface configuration. It can be a single value or a list. dns_search: example.com dns_search: - dc1.example.com - dc2.example.com domainname domainname declares a custom domain name to use for the service container. It must be a valid RFC 1123 hostname. entrypoint entrypoint declares the default entrypoint for the service container. This overrides the ENTRYPOINT instruction from the service's Dockerfile. If entrypoint is non-null, Compose ignores any default command from the image, for example the CMD instruction in the Dockerfile. See also command to set or override the default command to be executed by the entrypoint process. In its short form, the value can be defined as a string: entrypoint: /code/entrypoint.sh Alternatively, the value can also be a list, in a manner similar to the Dockerfile : entrypoint: - php - -d - zend_extension=/usr/local/lib/php/extensions/no-debug-non-zts-20100525/xdebug.so - -d - memory_limit=-1 - vendor/bin/phpunit If the value is null , the default entrypoint from the image is used. If the value is [] (empty list) or '' (empty string), the default entrypoint declared by the image is ignored, i.e. overridden to be empty. env_file env_file adds environment variables to the container based on the file content. env_file: .env env_file can also be a list. The files in the list are processed from the top down. For the same variable specified in two env files, the value from the last file in the list stands. env_file: - ./a.env - ./b.env List elements can also be declared as a mapping, which then lets you set an additional attribute required . This defaults to true . When required is set to false and the .env file is missing, Compose silently ignores the entry. env_file: - path: ./default.env required: true ## default - path: ./override.env required: false Relative path are resolved from the Compose file's parent folder. As absolute paths prevent the Compose file from being portable, Compose warns you when such a path is used to set env_file . Environment variables declared in the environment section override these values. This holds true even if those values are empty or undefined. Env_file format Each line in an .env file must be in VAR[=[VAL]] format. The following syntax rules apply: Lines beginning with # are processed as comments and ignored. Blank lines are ignored. Unquoted and double-quoted ( \" ) values have Interpolation applied. Each line represents a key-value pair. Values can optionally be quoted. VAR=VAL -> VAL VAR=\"VAL\" -> VAL VAR='VAL' -> VAL Inline comments for unquoted values must be preceded with a space. VAR=VAL ## comment -> VAL VAR=VAL## not a comment -> VAL## not a comment Inline comments for quoted values must follow the closing quote. VAR=\"VAL ## not a comment\" -> VAL ## not a comment VAR=\"VAL\" ## comment -> VAL Single-quoted ( ' ) values are used literally. VAR='$OTHER' -> $OTHER VAR='${OTHER}' -> ${OTHER} Quotes can be escaped with \\ . VAR='Let\\'s go!' -> Let's go! VAR=\"{\\\"hello\\\": \\\"json\\\"}\" -> {\"hello\": \"json\"} Common shell escape sequences including \\n , \\r , \\t , and \\\\ are supported in double-quoted values. VAR=\"some\\tvalue\" -> some value VAR='some\\tvalue' -> some\\tvalue VAR=some\\tvalue -> some\\tvalue VAL may be omitted, in such cases the variable value is an empty string. =VAL may be omitted, in such cases the variable is unset. ## Set Rails/Rack environment RACK_ENV=development VAR=\"quoted\" environment environment defines environment variables set in the container. environment can use either an array or a map. Any boolean values; true, false, yes, no, should be enclosed in quotes to ensure they are not converted to True or False by the YAML parser. Environment variables can be declared by a single key (no value to equals sign). In this case Compose relies on you to resolve the value. If the value is not resolved, the variable is unset and is removed from the service container environment. Map syntax: environment: RACK_ENV: development SHOW: \"true\" USER_INPUT: Array syntax: environment: - RACK_ENV=development - SHOW=true - USER_INPUT When both env_file and environment are set for a service, values set by environment have precedence. expose expose defines the (incoming) port or a range of ports that Compose exposes from the container. These ports must be accessible to linked services and should not be published to the host machine. Only the internal container ports can be specified. Syntax is /[] or /[] for a port range. When not explicitly set, tcp protocol is used. expose: - \"3000\" - \"8000\" - \"8080-8085/tcp Note If the Dockerfile for the image already exposes ports, it is visible to other containers on the network even if expose is not set in your Compose file. extends extends lets you share common configurations among different files, or even different projects entirely. With extends you can define a common set of service options in one place and refer to it from anywhere. You can refer to another Compose file and select a service you want to also use in your own application, with the ability to override some attributes for your own needs. You can use extends on any service together with other configuration keys. The extends value must be a mapping defined with a required service and an optional file key. extends: file: common.yml service: webapp service : Defines the name of the service being referenced as a base, for example web or database . file : The location of a Compose configuration file defining that service. Restrictions The following restrictions apply to the service being referenced: Services that have dependencies on other services cannot be used as a base. Therefore, any key that introduces a dependency on another service is incompatible with extends . The non-exhaustive list of such keys is: links , volumes_from , container mode (in ipc , pid , network_mode and net ), service mode (in ipc , pid and network_mode ), depends_on . Services cannot have circular references with extends . Compose returns an error in all of these cases. Finding referenced service file value can be: Not present. This indicates that another service within the same Compose file is being referenced. File path, which can be either: Relative path. This path is considered as relative to the location of the main Compose file. Absolute path. A service denoted by service must be present in the identified referenced Compose file. Compose returns an error if: The service denoted by service is not found. The Compose file denoted by file is not found. Merging service definitions Two service definitions, the main one in the current Compose file and the referenced one specified by extends , are merged in the following way: Mappings: Keys in mappings of the main service definition override keys in mappings of the referenced service definition. Keys that aren't overridden are included as is. Sequences: Items are combined together into a new sequence. The order of elements is preserved with the referenced items coming first and main items after. Scalars: Keys in the main service definition take precedence over keys in the referenced one. Mappings The following keys should be treated as mappings: annotations , build.args , build.labels , build.extra_hosts , deploy.labels , deploy.update_config , deploy.rollback_config , deploy.restart_policy , deploy.resources.limits , environment , healthcheck , labels , logging.options , sysctls , storage_opt , extra_hosts , ulimits . One exception that applies to healthcheck is that the main mapping cannot specify disable: true unless the referenced mapping also specifies disable: true . Compose returns an error in this case. For example, the input below: services: common: image: busybox environment: TZ: utc PORT: 80 cli: extends: service: common environment: PORT: 8080 Produces the following configuration for the cli service. The same output is produced if array syntax is used. environment: PORT: 8080 TZ: utc image: busybox Items under blkio_config.device_read_bps , blkio_config.device_read_iops , blkio_config.device_write_bps , blkio_config.device_write_iops , devices and volumes are also treated as mappings where key is the target path inside the container. For example, the input below: services: common: image: busybox volumes: - common-volume:/var/lib/backup/data:rw cli: extends: service: common volumes: - cli-volume:/var/lib/backup/data:ro Produces the following configuration for the cli service. Note that the mounted path now points to the new volume name and ro flag was applied. image: busybox volumes: - cli-volume:/var/lib/backup/data:ro If the referenced service definition contains extends mapping, the items under it are simply copied into the new merged definition. The merging process is then kicked off again until no extends keys are remaining. For example, the input below: services: base: image: busybox user: root common: image: busybox extends: service: base cli: extends: service: common Produces the following configuration for the cli service. Here, cli services gets user key from common service, which in turn gets this key from base service. image: busybox user: root Sequences The following keys should be treated as sequences: cap_add , cap_drop , configs , deploy.placement.constraints , deploy.placement.preferences , deploy.reservations.generic_resources , device_cgroup_rules , expose , external_links , ports , secrets , security_opt . Any duplicates resulting from the merge are removed so that the sequence only contains unique elements. For example, the input below: services: common: image: busybox security_opt: - label:role:ROLE cli: extends: service: common security_opt: - label:user:USER Produces the following configuration for the cli service. image: busybox security_opt: - label:role:ROLE - label:user:USER In case list syntax is used, the following keys should also be treated as sequences: dns , dns_search , env_file , tmpfs . Unlike sequence fields mentioned above, duplicates resulting from the merge are not removed. Scalars Any other allowed keys in the service definition should be treated as scalars. annotations annotations defines annotations for the container. annotations can use either an array or a map. annotations: com.example.foo: bar annotations: - com.example.foo=bar external_links external_links link service containers to services managed outside of your Compose application. external_links define the name of an existing service to retrieve using the platform lookup mechanism. An alias of the form SERVICE:ALIAS can be specified. external_links: - redis - database:mysql - database:postgresql extra_hosts extra_hosts adds hostname mappings to the container network interface configuration ( /etc/hosts for Linux). Short syntax Short syntax uses plain strings in a list. Values must set hostname and IP address for additional hosts in the form of HOSTNAME=IP . extra_hosts: - \"somehost=162.242.195.82\" - \"otherhost=50.31.209.229\" - \"myhostv6=::1\" IPv6 addresses can be enclosed in square brackets, for example: extra_hosts: - \"myhostv6=[::1]\" The separator = is preferred, but : can also be used. For example: extra_hosts: - \"somehost:162.242.195.82\" - \"myhostv6:::1\" Long syntax Alternatively, extra_hosts can be set as a mapping between hostname(s) and IP(s) extra_hosts: somehost: \"162.242.195.82\" otherhost: \"50.31.209.229\" myhostv6: \"::1\" Compose creates a matching entry with the IP address and hostname in the container's network configuration, which means for Linux /etc/hosts get extra lines: 162.242.195.82 somehost 50.31.209.229 otherhost ::1 myhostv6 group_add group_add specifies additional groups, by name or number, which the user inside the container must be a member of. An example of where this is useful is when multiple containers (running as different users) need to all read or write the same file on a shared volume. That file can be owned by a group shared by all the containers, and specified in group_add . services: myservice: image: alpine group_add: - mail Running id inside the created container must show that the user belongs to the mail group, which would not have been the case if group_add were not declared. healthcheck healthcheck declares a check that's run to determine whether or not the service containers are \"healthy\". It works in the same way, and has the same default values, as the HEALTHCHECK Dockerfile instruction set by the service's Docker image. Your Compose file can override the values set in the Dockerfile. healthcheck: test: [\"CMD\", \"curl\", \"-f\", \"http://localhost\"] interval: 1m30s timeout: 10s retries: 3 start_period: 40s start_interval: 5s interval , timeout , start_period , and start_interval are specified as durations . test defines the command Compose runs to check container health. It can be either a string or a list. If it's a list, the first item must be either NONE , CMD or CMD-SHELL . If it's a string, it's equivalent to specifying CMD-SHELL followed by that string. ## Hit the local web app test: [\"CMD\", \"curl\", \"-f\", \"http://localhost\"] Using CMD-SHELL runs the command configured as a string using the container's default shell ( /bin/sh for Linux). Both forms below are equivalent: test: [\"CMD-SHELL\", \"curl -f http://localhost || exit 1\"] test: curl -f https://localhost || exit 1 NONE disables the healthcheck, and is mostly useful to disable the Healthcheck Dockerfile instruction set by the service's Docker image. Alternatively, the healthcheck set by the image can be disabled by setting disable: true : healthcheck: disable: true hostname hostname declares a custom host name to use for the service container. It must be a valid RFC 1123 hostname. image image specifies the image to start the container from. image must follow the Open Container Specification addressable image format , as [/][/][:|@] . image: redis image: redis:5 image: redis@sha256:0ed5d5928d4737458944eb604cc8509e245c3e19d02ad83935398bc4b991aac7 image: library/redis image: docker.io/library/redis image: my_private.registry:5000/redis If the image does not exist on the platform, Compose attempts to pull it based on the pull_policy . If you are also using the Compose Build Specification , there are alternative options for controlling the precedence of pull over building the image from source, however pulling the image is the default behavior. image may be omitted from a Compose file as long as a build section is declared. If you are not using the Compose Build Specification, Compose won't work if image is missing from the Compose file. init init runs an init process (PID 1) inside the container that forwards signals and reaps processes. Set this option to true to enable this feature for the service. services: web: image: alpine:latest init: true The init binary that is used is platform specific. ipc ipc configures the IPC isolation mode set by the service container. Available values are platform specific, but Compose defines specific values which must be implemented as described if supported: shareable : Gives the container its own private IPC namespace, with a possibility to share it with other containers. service:{name} : Makes the container join another container's ( shareable ) IPC namespace. ipc: \"shareable\" ipc: \"service:[service name]\" uts uts configures the UTS namespace mode set for the service container. When unspecified it is the runtime's decision to assign a UTS namespace, if supported. Available values are: 'host' : Results in the container using the same UTS namespace as the host. uts: \"host\" isolation isolation specifies a container\u2019s isolation technology. Supported values are platform specific. labels labels add metadata to containers. You can use either an array or a map. It's recommended that you use reverse-DNS notation to prevent your labels from conflicting with those used by other software. labels: com.example.description: \"Accounting webapp\" com.example.department: \"Finance\" com.example.label-with-empty-value: \"\" labels: - \"com.example.description=Accounting webapp\" - \"com.example.department=Finance\" - \"com.example.label-with-empty-value\" Compose creates containers with canonical labels: com.docker.compose.project set on all resources created by Compose to the user project name com.docker.compose.service set on service containers with service name as defined in the Compose file The com.docker.compose label prefix is reserved. Specifying labels with this prefix in the Compose file results in a runtime error. links Note Availability of the links attribute is implementation specific. links defines a network link to containers in another service. Either specify both the service name and a link alias ( SERVICE:ALIAS ), or just the service name. web: links: - db - db:database - redis Containers for the linked service are reachable at a hostname identical to the alias, or the service name if no alias is specified. Links are not required to enable services to communicate. When no specific network configuration is set, any service is able to reach any other service at that service\u2019s name on the default network. If services do declare networks they are attached to, links does not override the network configuration and services not attached to a shared network are not be able to communicate. Compose doesn't warn you about a configuration mismatch. Links also express implicit dependency between services in the same way as depends_on , so they determine the order of service startup. logging logging defines the logging configuration for the service. logging: driver: syslog options: syslog-address: \"tcp://192.168.0.42:123\" The driver name specifies a logging driver for the service's containers. The default and available values are platform specific. Driver specific options can be set with options as key-value pairs. network_mode network_mode sets a service container's network mode. Available values are platform specific, but Compose defines specific values which must be implemented as described if supported: none : Turns off all container networking. host : Gives the container raw access to the host's network interface. service:{name} : Gives the containers access to the specified service only. network_mode: \"host\" network_mode: \"none\" network_mode: \"service:[service name]\" When set, the networks attribute is not allowed and Compose rejects any Compose file containing both attributes. networks networks defines the networks that service containers are attached to, referencing entries under the top-level networks key . services: some-service: networks: - some-network - other-network aliases aliases declares alternative hostnames for the service on the network. Other containers on the same network can use either the service name or an alias to connect to one of the service's containers. Since aliases are network-scoped, the same service can have different aliases on different networks. Note A network-wide alias can be shared by multiple containers, and even by multiple services. If it is, then exactly which container the name resolves to is not guaranteed. services: some-service: networks: some-network: aliases: - alias1 - alias3 other-network: aliases: - alias2 In the following example, service frontend is able to reach the backend service at the hostname backend or database on the back-tier network. The service monitoring is able to reach same backend service at backend or mysql on the admin network. services: frontend: image: example/webapp networks: - front-tier - back-tier monitoring: image: example/monitoring networks: - admin backend: image: example/backend networks: back-tier: aliases: - database admin: aliases: - mysql networks: front-tier: back-tier: admin: ipv4_address, ipv6_address Specify a static IP address for a service container when joining the network. The corresponding network configuration in the top-level networks section must have an ipam attribute with subnet configurations covering each static address. services: frontend: image: example/webapp networks: front-tier: ipv4_address: 172.16.238.10 ipv6_address: 2001:3984:3989::10 networks: front-tier: ipam: driver: default config: - subnet: \"172.16.238.0/24\" - subnet: \"2001:3984:3989::/64\" link_local_ips link_local_ips specifies a list of link-local IPs. Link-local IPs are special IPs which belong to a well known subnet and are purely managed by the operator, usually dependent on the architecture where they are deployed. Implementation is platform specific. Example: services: app: image: busybox command: top networks: app_net: link_local_ips: - 57.123.22.11 - 57.123.22.13 networks: app_net: driver: bridge mac_address mac_address sets the MAC address used by the service container when connecting to this particular network. priority priority indicates in which order Compose connects the service\u2019s containers to its networks. If unspecified, the default value is 0. In the following example, the app service connects to app_net_1 first as it has the highest priority. It then connects to app_net_3 , then app_net_2 , which uses the default priority value of 0. services: app: image: busybox command: top networks: app_net_1: priority: 1000 app_net_2: app_net_3: priority: 100 networks: app_net_1: app_net_2: app_net_3: mac_address mac_address sets a MAC address for the service container. Note Container runtimes might reject this value (ie. Docker Engine >= v25.0). In that case, you should use networks.mac_address instead. mem_limit DEPRECATED: use deploy.limits.memory mem_reservation DEPRECATED: use deploy.reservations.memory mem_swappiness mem_swappiness defines as a percentage, a value between 0 and 100, for the host kernel to swap out anonymous memory pages used by a container. 0 : Turns off anonymous page swapping. 100 : Sets all anonymous pages as swappable. The default value is platform specific. memswap_limit memswap_limit defines the amount of memory the container is allowed to swap to disk. This is a modifier attribute that only has meaning if memory is also set. Using swap lets the container write excess memory requirements to disk when the container has exhausted all the memory that is available to it. There is a performance penalty for applications that swap memory to disk often. If memswap_limit is set to a positive integer, then both memory and memswap_limit must be set. memswap_limit represents the total amount of memory and swap that can be used, and memory controls the amount used by non-swap memory. So if memory =\"300m\" and memswap_limit =\"1g\", the container can use 300m of memory and 700m (1g - 300m) swap. If memswap_limit is set to 0, the setting is ignored, and the value is treated as unset. If memswap_limit is set to the same value as memory , and memory is set to a positive integer, the container does not have access to swap. If memswap_limit is unset, and memory is set, the container can use as much swap as the memory setting, if the host container has swap memory configured. For instance, if memory =\"300m\" and memswap_limit is not set, the container can use 600m in total of memory and swap. If memswap_limit is explicitly set to -1, the container is allowed to use unlimited swap, up to the amount available on the host system. oom_kill_disable If oom_kill_disable is set, Compose configures the platform so it won't kill the container in case of memory starvation. oom_score_adj oom_score_adj tunes the preference for containers to be killed by platform in case of memory starvation. Value must be within [-1000,1000] range. pid pid sets the PID mode for container created by Compose. Supported values are platform specific. pids_limit DEPRECATED: use deploy.resources.limits.pids pids_limit tunes a container\u2019s PIDs limit. Set to -1 for unlimited PIDs. pids_limit: 10 platform platform defines the target platform the containers for the service run on. It uses the os[/arch[/variant]] syntax. The values of os , arch , and variant must conform to the convention used by the OCI Image Spec . Compose uses this attribute to determine which version of the image is pulled and/or on which platform the service\u2019s build is performed. platform: darwin platform: windows/amd64 platform: linux/arm64/v8 ports Exposes container ports. Note Port mapping must not be used with network_mode: host otherwise a runtime error occurs. Short syntax The short syntax is a colon-separated string to set the host IP, host port, and container port in the form: [HOST:]CONTAINER[/PROTOCOL] where: HOST is [IP:](port | range) CONTAINER is port | range PROTOCOL to restrict port to specified protocol. tcp and udp values are defined by the Specification, Compose offers support for platform-specific protocol names. If host IP is not set, it binds to all network interfaces. Ports can be either a single value or a range. Host and container must use equivalent ranges. Either specify both ports ( HOST:CONTAINER ), or just the container port. In the latter case, the container runtime automatically allocates any unassigned port of the host. HOST:CONTAINER should always be specified as a (quoted) string, to avoid conflicts with yaml base-60 float . Examples: ports: - \"3000\" - \"3000-3005\" - \"8000:8000\" - \"9090-9091:8080-8081\" - \"49100:22\" - \"8000-9000:80\" - \"127.0.0.1:8001:8001\" - \"127.0.0.1:5000-5010:5000-5010\" - \"6060:6060/udp\" Note If Host IP mapping is not supported by a container engine, Compose rejects the Compose file and ignores the specified host IP. Long syntax The long form syntax allows the configuration of additional fields that can't be expressed in the short form. target : The container port published : The publicly exposed port. It is defined as a string and can be set as a range using syntax start-end . It means the actual port is assigned a remaining available port, within the set range. host_ip : The Host IP mapping, unspecified means all network interfaces ( 0.0.0.0 ). protocol : The port protocol ( tcp or udp ). Defaults to tcp . mode : host : For publishing a host port on each node, or ingress for a port to be load balanced. Defaults to ingress . name : A human-readable name for the port, used to document it's usage within the service ports: - name: http target: 80 host_ip: 127.0.0.1 published: \"8080\" protocol: tcp mode: host - name: https target: 443 host_ip: 127.0.0.1 published: \"8083-9000\" protocol: tcp mode: host privileged privileged configures the service container to run with elevated privileges. Support and actual impacts are platform specific. profiles profiles defines a list of named profiles for the service to be enabled under. If unassigned, the service is always started but if assigned, it is only started if the profile is activated. If present, profiles follow the regex format of [a-zA-Z0-9][a-zA-Z0-9_.-]+ . services: frontend: image: frontend profiles: [\"frontend\"] phpmyadmin: image: phpmyadmin depends_on: - db profiles: - debug pull_policy pull_policy defines the decisions Compose makes when it starts to pull images. Possible values are: always : Compose always pulls the image from the registry. never : Compose doesn't pull the image from a registry and relies on the platform cached image. If there is no cached image, a failure is reported. missing : Compose pulls the image only if it's not available in the platform cache. This is the default option if you are not also using the Compose Build Specification . if_not_present is considered an alias for this value for backward compatibility. build : Compose builds the image. Compose rebuilds the image if it's already present. read_only read_only configures the service container to be created with a read-only filesystem. restart restart defines the policy that the platform applies on container termination. no : The default restart policy. It does not restart the container under any circumstances. always : The policy always restarts the container until its removal. on-failure : The policy restarts the container if the exit code indicates an error. unless-stopped : The policy restarts the container irrespective of the exit code but stops restarting when the service is stopped or removed. restart: \"no\" restart: always restart: on-failure restart: unless-stopped runtime runtime specifies which runtime to use for the service\u2019s containers. The value of runtime is specific to the implementation. For example, runtime can be the name of an implementation of OCI Runtime Spec , such as \"runc\". web: image: busybox:latest command: true runtime: runc scale scale specifies the default number of containers to deploy for this service. When both are set, scale must be consistent with the replicas attribute in the Deploy Specification . secrets secrets grants access to sensitive data defined by secrets on a per-service basis. Two different syntax variants are supported; the short syntax and the long syntax. Compose reports an error if the secret doesn't exist on the platform or isn't defined in the secrets section of the Compose file. Services can be granted access to multiple secrets. Long and short syntax for secrets may be used in the same Compose file. Defining a secret in the top-level secrets must not imply granting any service access to it. Such grant must be explicit within service specification as secrets service element. Short syntax The short syntax variant only specifies the secret name. This grants the container access to the secret and mounts it as read-only to /run/secrets/ within the container. The source name and destination mountpoint are both set to the secret name. The following example uses the short syntax to grant the frontend service access to the server-certificate secret. The value of server-certificate is set to the contents of the file ./server.cert . services: frontend: image: example/webapp secrets: - server-certificate secrets: server-certificate: file: ./server.cert Long syntax The long syntax provides more granularity in how the secret is created within the service's containers. source : The name of the secret as it exists on the platform. target : The name of the file to be mounted in /run/secrets/ in the service's task container, or absolute path of the file if an alternate location is required. Defaults to source if not specified. uid and gid : The numeric UID or GID that owns the file within /run/secrets/ in the service's task containers. Default value is USER running container. mode : The permissions for the file to be mounted in /run/secrets/ in the service's task containers, in octal notation. The default value is world-readable permissions (mode 0444 ). The writable bit must be ignored if set. The executable bit may be set. Note that the uid , gid , and mode attributes are implementation specific. The following example sets the name of the server-certificate secret file to server.crt within the container, sets the mode to 0440 (group-readable), and sets the user and group to 103 . The value of server-certificate secret is provided by the platform through a lookup and the secret's lifecycle is not directly managed by Compose. services: frontend: image: example/webapp secrets: - source: server-certificate target: server.cert uid: \"103\" gid: \"103\" mode: 0440 secrets: server-certificate: external: true security_opt security_opt overrides the default labeling scheme for each container. security_opt: - label:user:USER - label:role:ROLE For further default labeling schemes you can override, see Security configuration . shm_size shm_size configures the size of the shared memory ( /dev/shm partition on Linux) allowed by the service container. It's specified as a byte value . stdin_open stdin_open configures a service containers to run with an allocated stdin. stop_grace_period stop_grace_period specifies how long Compose must wait when attempting to stop a container if it doesn't handle SIGTERM (or whichever stop signal has been specified with stop_signal ), before sending SIGKILL. It's specified as a duration . stop_grace_period: 1s stop_grace_period: 1m30s Default value is 10 seconds for the container to exit before sending SIGKILL. stop_signal stop_signal defines the signal that Compose uses to stop the service containers. If unset containers are stopped by Compose by sending SIGTERM . stop_signal: SIGUSR1 storage_opt storage_opt defines storage driver options for a service. storage_opt: size: '1G' sysctls sysctls defines kernel parameters to set in the container. sysctls can use either an array or a map. sysctls: net.core.somaxconn: 1024 net.ipv4.tcp_syncookies: 0 sysctls: - net.core.somaxconn=1024 - net.ipv4.tcp_syncookies=0 You can only use sysctls that are namespaced in the kernel. Docker does not support changing sysctls inside a container that also modify the host system. For an overview of supported sysctls, refer to configure namespaced kernel parameters (sysctls) at runtime . tmpfs tmpfs mounts a temporary file system inside the container. It can be a single value or a list. tmpfs: /run tmpfs: - /run - /tmp tty tty configures service container to run with a TTY. ulimits ulimits overrides the default ulimits for a container. It's specified either as an integer for a single limit or as mapping for soft/hard limits. ulimits: nproc: 65535 nofile: soft: 20000 hard: 40000 user user overrides the user used to run the container process. The default is set by the image (i.e. Dockerfile USER ). If it's not set, then root . userns_mode userns_mode sets the user namespace for the service. Supported values are platform specific and may depend on platform configuration. userns_mode: \"host\" volumes volumes define mount host paths or named volumes that are accessible by service containers. You can use volumes to define multiple types of mounts; volume , bind , tmpfs , or npipe . If the mount is a host path and is only used by a single service, it can be declared as part of the service definition. To reuse a volume across multiple services, a named volume must be declared in the top-level volumes key . The following example shows a named volume ( db-data ) being used by the backend service, and a bind mount defined for a single service. services: backend: image: example/backend volumes: - type: volume source: db-data target: /data volume: nocopy: true - type: bind source: /var/run/postgres/postgres.sock target: /var/run/postgres/postgres.sock volumes: db-data: Short syntax The short syntax uses a single string with colon-separated values to specify a volume mount ( VOLUME:CONTAINER_PATH ), or an access mode ( VOLUME:CONTAINER_PATH:ACCESS_MODE ). VOLUME : Can be either a host path on the platform hosting containers (bind mount) or a volume name. CONTAINER_PATH : The path in the container where the volume is mounted. ACCESS_MODE : A comma-separated , list of options: rw : Read and write access. This is the default if none is specified. ro : Read-only access. z : SELinux option indicating that the bind mount host content is shared among multiple containers. Z : SELinux option indicating that the bind mount host content is private and unshared for other containers. Note The SELinux re-labeling bind mount option is ignored on platforms without SELinux. Note Relative host paths are only supported by Compose that deploy to a local container runtime. This is because the relative path is resolved from the Compose file\u2019s parent directory which is only applicable in the local case. When Compose deploys to a non-local platform it rejects Compose files which use relative host paths with an error. To avoid ambiguities with named volumes, relative paths should always begin with . or .. . Long syntax The long form syntax allows the configuration of additional fields that can't be expressed in the short form. type : The mount type. Either volume , bind , tmpfs , npipe , or cluster source : The source of the mount, a path on the host for a bind mount, or the name of a volume defined in the top-level volumes key . Not applicable for a tmpfs mount. target : The path in the container where the volume is mounted. read_only : Flag to set the volume as read-only. bind : Used to configure additional bind options: propagation : The propagation mode used for the bind. create_host_path : Creates a directory at the source path on host if there is nothing present. Compose does nothing if there is something present at the path. This is automatically implied by short syntax for backward compatibility with docker-compose legacy. selinux : The SELinux re-labeling option z (shared) or Z (private) volume : Configures additional volume options: nocopy : Flag to disable copying of data from a container when a volume is created. tmpfs : Configures additional tmpfs options: size : The size for the tmpfs mount in bytes (either numeric or as bytes unit). mode : The file mode for the tmpfs mount as Unix permission bits as an octal number. consistency : The consistency requirements of the mount. Available values are platform specific. volumes_from volumes_from mounts all of the volumes from another service or container. You can optionally specify read-only access ro or read-write rw . If no access level is specified, then read-write access is used. You can also mount volumes from a container that is not managed by Compose by using the container: prefix. volumes_from: - service_name - service_name:ro - container:container_name - container:container_name:rw working_dir working_dir overrides the container's working directory which is specified by the image, for example Dockerfile's WORKDIR . Networks top-level element Networks are the layer that allow services to communicate with each other. The top-level networks element lets you configure named networks that can be reused across multiple services. To use a network across multiple services, you must explicitly grant each service access by using the networks attribute within the services top-level element. The networks top-level element has additional syntax that provides more granular control. Examples Basic example In the following example, at runtime, networks front-tier and back-tier are created and the frontend service is connected to front-tier and back-tier networks. services: frontend: image: example/webapp networks: - front-tier - back-tier networks: front-tier: back-tier: Advanced example services: proxy: build: ./proxy networks: - frontend app: build: ./app networks: - frontend - backend db: image: postgres networks: - backend networks: frontend: ## Use a custom driver driver: custom-driver-1 backend: ## Use a custom driver which takes special options driver: custom-driver-2 driver_opts: foo: \"1\" bar: \"2\" The advanced example shows a Compose file which defines two custom networks. The proxy service is isolated from the db service, because they do not share a network in common. Only app can talk to both. Attributes driver driver specifies which driver should be used for this network. Compose returns an error if the driver is not available on the platform. networks: db-data: driver: overlay Default and available values are platform specific. Compose supports the following drivers: none and host host : Use the host's networking stack. none : Turn off networking. host or none The syntax for using built-in networks such as host and none is different, as such networks implicitly exist outside the scope of Compose. To use them, you must define an external network with the name host or none and an alias that Compose can use ( hostnet and nonet in the following example), then grant the service access to that network using its alias. services: web: networks: hostnet: {} networks: hostnet: external: true name: host services: web: ... networks: nonet: {} networks: nonet: external: true name: none driver_opts driver_opts specifies a list of options as key-value pairs to pass to the driver. These options are driver-dependent. Consult the driver's documentation for more information. networks: db-data: driver_opts: foo: \"bar\" baz: 1 attachable If attachable is set to true , then standalone containers should be able to attach to this network, in addition to services. If a standalone container attaches to the network, it can communicate with services and other standalone containers that are also attached to the network. networks: mynet1: driver: overlay attachable: true enable_ipv6 enable_ipv6 enables IPv6 networking. For an example, see step four of Create an IPv6 network . external If set to true : - external specifies that this network\u2019s lifecycle is maintained outside of that of the application. Compose doesn't attempt to create these networks, and returns an error if one doesn't exist. - All other attributes apart from name are irrelevant. If Compose detects any other attribute, it rejects the Compose file as invalid. In the example below, proxy is the gateway to the outside world. Instead of attempting to create a network, Compose queries the platform for an existing network simply called outside and connects the proxy service's containers to it. services: proxy: image: example/proxy networks: - outside - default app: image: example/app networks: - default networks: outside: external: true ipam ipam specifies a custom IPAM configuration. This is an object with several properties, each of which is optional: driver : Custom IPAM driver, instead of the default. config : A list with zero or more configuration elements, each containing a: subnet : Subnet in CIDR format that represents a network segment ip_range : Range of IPs from which to allocate container IPs gateway : IPv4 or IPv6 gateway for the master subnet aux_addresses : Auxiliary IPv4 or IPv6 addresses used by Network driver, as a mapping from hostname to IP options : Driver-specific options as a key-value mapping. networks: mynet1: ipam: driver: default config: - subnet: 172.28.0.0/16 ip_range: 172.28.5.0/24 gateway: 172.28.5.254 aux_addresses: host1: 172.28.1.5 host2: 172.28.1.6 host3: 172.28.1.7 options: foo: bar baz: \"0\" internal By default, Compose provides external connectivity to networks. internal , when set to true , allows you to create an externally isolated network. labels Add metadata to containers using labels . You can use either an array or a dictionary. It is recommended that you use reverse-DNS notation to prevent labels from conflicting with those used by other software. networks: mynet1: labels: com.example.description: \"Financial transaction network\" com.example.department: \"Finance\" com.example.label-with-empty-value: \"\" networks: mynet1: labels: - \"com.example.description=Financial transaction network\" - \"com.example.department=Finance\" - \"com.example.label-with-empty-value\" Compose sets com.docker.compose.project and com.docker.compose.network labels. name name sets a custom name for the network. The name field can be used to reference networks which contain special characters. The name is used as is and is not scoped with the project name. networks: network1: name: my-app-net It can also be used in conjunction with the external property to define the platform network that Compose should retrieve, typically by using a parameter so the Compose file doesn't need to hard-code runtime specific values: networks: network1: external: true name: \"${NETWORK_ID}\" Volumes top-level element Volumes are persistent data stores implemented by the container engine. Compose offers a neutral way for services to mount volumes, and configuration parameters to allocate them to infrastructure. The top-level volumes declaration lets you configure named volumes that can be reused across multiple services. To use a volume across multiple services, you must explicitly grant each service access by using the volumes attribute within the services top-level element. The volumes attribute has additional syntax that provides more granular control. Example The following example shows a two-service setup where a database's data directory is shared with another service as a volume, named db-data , so that it can be periodically backed up. services: backend: image: example/database volumes: - db-data:/etc/data backup: image: backup-service volumes: - db-data:/var/lib/backup/data volumes: db-data: The db-data volume is mounted at the /var/lib/backup/data and /etc/data container paths for backup and backend respectively. Running docker compose up creates the volume if it doesn't already exist. Otherwise, the existing volume is used and is recreated if it's manually deleted outside of Compose. Attributes An entry under the top-level volumes section can be empty, in which case it uses the container engine's default configuration for creating a volume. Optionally, you can configure it with the following keys: driver Specifies which volume driver should be used. Default and available values are platform specific. If the driver is not available, Compose returns an error and doesn't deploy the application. volumes: db-data: driver: foobar driver_opts driver_opts specifies a list of options as key-value pairs to pass to the driver for this volume. The options are driver-dependent. volumes: example: driver_opts: type: \"nfs\" o: \"addr=10.40.0.199,nolock,soft,rw\" device: \":/docker/example\" external If set to true : - external specifies that this volume already exists on the platform and its lifecycle is managed outside of that of the application. Compose doesn't then create the volume, and returns an error if the volume doesn't exist. - All other attributes apart from name are irrelevant. If Compose detects any other attribute, it rejects the Compose file as invalid. In the example below, instead of attempting to create a volume called {project_name}_db-data , Compose looks for an existing volume simply called db-data and mounts it into the backend service's containers. services: backend: image: example/database volumes: - db-data:/etc/data volumes: db-data: external: true labels labels are used to add metadata to volumes. You can use either an array or a dictionary. It's recommended that you use reverse-DNS notation to prevent your labels from conflicting with those used by other software. volumes: db-data: labels: com.example.description: \"Database volume\" com.example.department: \"IT/Ops\" com.example.label-with-empty-value: \"\" volumes: db-data: labels: - \"com.example.description=Database volume\" - \"com.example.department=IT/Ops\" - \"com.example.label-with-empty-value\" Compose sets com.docker.compose.project and com.docker.compose.volume labels. name name sets a custom name for a volume. The name field can be used to reference volumes that contain special characters. The name is used as is and is not scoped with the stack name. volumes: db-data: name: \"my-app-data\" This makes it possible to make this lookup name a parameter of the Compose file, so that the model ID for the volume is hard-coded but the actual volume ID on the platform is set at runtime during deployment. For example, if DATABASE_VOLUME=my_volume_001 in your .env file: volumes: db-data: name: ${DATABASE_VOLUME} Running docker compose up uses the volume called my_volume_001 . It can also be used in conjunction with the external property. This means the name of the volume used to lookup the actual volume on the platform is set separately from the name used to refer to it within the Compose file: volumes: db-data: external: name: actual-name-of-volume Configs top-level element Configs allow services to adapt their behaviour without the need to rebuild a Docker image. Services can only access configs when explicitly granted by a configs attribute within the services top-level element. As with volumes, configs are mounted as files into a service's container's filesystem. The location of the mount point within the container defaults to / in Linux containers and C:\\ in Windows containers. By default, the config: - Is owned by the user running the container command but can be overridden by service configuration. - Has world-readable permissions (mode 0444), unless the service is configured to override this. The top-level configs declaration defines or references configuration data that is granted to services in your Compose application. The source of the config is either file or external . file : The config is created with the contents of the file at the specified path. environment : The config content is created with the value of an environment variable. content : The content is created with the inlined value. external : If set to true, external specifies that this config has already been created. Compose does not attempt to create it, and if it does not exist, an error occurs. name : The name of the config object in the container engine to look up. This field can be used to reference configs that contain special characters. The name is used as is and will not be scoped with the project name. Example 1 _http_config is created when the application is deployed, by registering the content of the httpd.conf as the configuration data. configs: http_config: file: ./httpd.conf Alternatively, http_config can be declared as external. Compose looks up http_config to expose the configuration data to relevant services. configs: http_config: external: true Example 2 _app_config is created when the application is deployed, by registering the inlined content as the configuration data. This comes with the benefits Compose will infer variables when creating the config, which allows to adjust content according to service configuration: configs: app_config: content: | debug=${DEBUG} spring.application.admin.enabled=${DEBUG} spring.application.name=${COMPOSE_PROJECT_NAME} Example 3 External configs lookup can also use a distinct key by specifying a name . The following example modifies the previous one to look up a config using the parameter HTTP_CONFIG_KEY . The the actual lookup key will is set at deployment time by the interpolation of variables, but exposed to containers as hard-coded ID http_config . configs: http_config: external: true name: \"${HTTP_CONFIG_KEY}\" If external is set to true , all other attributes apart from name are irrelevant. If Compose detecs any other attribute, it rejects the Compose file as invalid. Secrets top-level element Secrets are a flavor of Configs focusing on sensitive data, with specific constraint for this usage. Services can only access secrets when explicitly granted by a secrets attribute within the services top-level element. The top-level secrets declaration defines or references sensitive data that is granted to the services in your Compose application. The source of the secret is either file or environment . file : The secret is created with the contents of the file at the specified path. environment : The secret is created with the value of an environment variable. external : If set to true, external specifies that this secret has already been created. Compose does not attempt to create it, and if it does not exist, an error occurs. name : The name of the secret object in Docker. This field can be used to reference secrets that contain special characters. The name is used as is and isn't scoped with the project name. Example 1 server-certificate secret is created as _server-certificate when the application is deployed, by registering content of the server.cert as a platform secret. secrets: server-certificate: file: ./server.cert Example 2 token secret is created as _token when the application is deployed, by registering the content of the OAUTH_TOKEN environment variable as a platform secret. secrets: token: environment: \"OAUTH_TOKEN\" Alternatively, server-certificate can be declared as external. Compose looks up the server-certificate secret to expose to relevant services. secrets: server-certificate: external: true Example 3 External secrets lookup can also use a distinct key by specifying a name . The following example modifies the previous example to look up a secret using the name CERTIFICATE_KEY . The actual lookup key is set at deployment time by the interpolation of variables, but exposed to containers as hard-coded ID server-certificate . secrets: server-certificate: external: true name: \"${CERTIFICATE_KEY}\" If external is set to true , all other attributes apart from name are irrelevant. If Compose detects any other attribute, it rejects the Compose file as invalid. Your Compose file needs to explicitly grant access to the secrets to relevant services in your application. Fragments With Compose, you can use built-in YAML features to make your Compose file neater and more efficient. Anchors and aliases let you create re-usable blocks. This is useful if you start to find common configurations that span multiple services. Having re-usable blocks minimizes potential mistakes. Anchors are created using the & sign. The sign is followed by an alias name. You can use this alias with the * sign later to reference the value following the anchor. Make sure there is no space between the & and the * characters and the following alias name. You can use more than one anchor and alias in a single Compose file. Example 1 volumes: db-data: &default-volume driver: default metrics: *default-volume In the example above, a default-volume anchor is created based on the db-data volume. It is later reused by the alias *default-volume to define the metrics volume. Anchor resolution takes place before variables interpolation , so variables can't be used to set anchors or aliases. Example 2 services: first: image: my-image:latest environment: &env - CONFIG_KEY - EXAMPLE_KEY - DEMO_VAR second: image: another-image:latest environment: *env If you have an anchor that you want to use in more than one service, use it in conjunction with an extension to make your Compose file easier to maintain. Example 3 You may want to partially override values. Compose follows the rule outlined by YAML merge type . In the following example, metrics volume specification uses alias to avoid repetition but overrides name attribute: services: backend: image: example/database volumes: - db-data - metrics volumes: db-data: &default-volume driver: default name: \"data\" metrics: <<: *default-volume name: \"metrics\" Example 4 You can also extend the anchor to add additional values. services: first: image: my-image:latest environment: &env FOO: BAR ZOT: QUIX second: image: another-image:latest environment: <<: *env YET_ANOTHER: VARIABLE Note YAML merge only applies to mappings, and can't be used with sequences. In example above, the environment variables must be declared using the FOO: BAR mapping syntax, while the sequence syntax - FOO=BAR is only valid when no fragments are involved. Extension As with Fragments , Extensions can be used to make your Compose file more efficient and easier to maintain. Extensions can also be used with anchors and aliases . Use the prefix x- as a top-level element to modularize configurations that you want to reuse. Compose ignores any fields that start with x- , this is the sole exception where Compose silently ignores unrecognized fields. They also can be used within any structure in a Compose file where user-defined keys are not expected. Compose use those to enable experimental features, the same way browsers add support for custom CSS features Example 1 x-custom: foo: - bar - zot services: webapp: image: example/webapp x-foo: bar service: backend: deploy: placement: x-aws-role: \"arn:aws:iam::XXXXXXXXXXXX:role/foo\" x-aws-region: \"eu-west-3\" x-azure-region: \"france-central\" Example 2 x-env: &env environment: - CONFIG_KEY - EXAMPLE_KEY services: first: <<: *env image: my-image:latest second: <<: *env image: another-image:latest In this example, the environment variables do not belong to either of the services. They\u2019ve been lifted out completely into the x-env extension field. This defines a new node which contains the environment field. The &env YAML anchor is used so both services can reference the extension field\u2019s value as *env . Example 3 x-function: &function labels: function: \"true\" depends_on: - gateway networks: - functions deploy: placement: constraints: - 'node.platform.os == linux' services: ## Node.js gives OS info about the node (Host) nodeinfo: <<: *function image: functions/nodeinfo:latest environment: no_proxy: \"gateway\" https_proxy: $https_proxy ## Uses `cat` to echo back response, fastest function to execute. echoit: <<: *function image: functions/alpine:health environment: fprocess: \"cat\" no_proxy: \"gateway\" https_proxy: $https_proxy The nodeinfo and echoit services both include the x-function extension via the &function anchor, then set their specific image and environment. Example 4 Using YAML merge it is also possible to use multiple extensions and share and override additional attributes for specific needs: x-environment: &default-environment FOO: BAR ZOT: QUIX x-keys: &keys KEY: VALUE services: frontend: image: example/webapp environment: << : [*default-environment, *keys] YET_ANOTHER: VARIABLE Note YAML merge only applies to mappings, and can't be used with sequences. In the example above, the environment variables are declared using the FOO: BAR mapping syntax, while the sequence syntax - FOO=BAR is only valid when no fragments are involved. Informative Historical Notes This section is informative. At the time of writing, the following prefixes are known to exist: Prefix Vendor/Organization docker Docker kubernetes Kubernetes Specifying byte values Values express a byte value as a string in {amount}{byte unit} format: The supported units are b (bytes), k or kb (kilo bytes), m or mb (mega bytes) and g or gb (giga bytes). 2b 1024kb 2048k 300m 1gb Specifying durations Values express a duration as a string in the form of {value}{unit} . The supported units are us (microseconds), ms (milliseconds), s (seconds), m (minutes) and h (hours). Values can combine multiple values without separator. 10ms 40s 1m30s 1h5m30s20ms Interpolation Values in a Compose file can be set by variables and interpolated at runtime. Compose files use a Bash-like syntax ${VARIABLE} . Both $VARIABLE and ${VARIABLE} syntax is supported. Default values can be defined inline using typical shell syntax: ${VARIABLE:-default} evaluates to default if VARIABLE is unset or empty in the environment. ${VARIABLE-default} evaluates to default only if VARIABLE is unset in the environment. Similarly, the following syntax allows you to specify mandatory variables: ${VARIABLE:?err} exits with an error message containing err if VARIABLE is unset or empty in the environment. ${VARIABLE?err} exits with an error message containing err only if VARIABLE is unset in the environment. Interpolation can also be nested: ${VARIABLE:-${FOO}} ${VARIABLE?$FOO} ${VARIABLE:-${FOO:-default}} Other extended shell-style features, such as ${VARIABLE/foo/bar} , are not supported by Compose. You can use a $$ (double-dollar sign) when your configuration needs a literal dollar sign. This also prevents Compose from interpolating a value, so a $$ allows you to refer to environment variables that you don't want processed by Compose. web: build: . command: \"$$VAR_NOT_INTERPOLATED_BY_COMPOSE\" If Compose can't resolve a substituted variable and no default value is defined, it displays a warning and substitutes the variable with an empty string. As any values in a Compose file can be interpolated with variable substitution, including compact string notation for complex elements, interpolation is applied before a merge on a per-file basis. Interpolation applies only to YAML values, not to keys. For the few places where keys are actually arbitrary user-defined strings, such as labels or environment , an alternate equal sign syntax must be used for interpolation to apply. For example: services: foo: labels: \"$VAR_NOT_INTERPOLATED_BY_COMPOSE\": \"BAR\" services: foo: labels: - \"$VAR_INTERPOLATED_BY_COMPOSE=BAR\" Merge and override Compose lets you define a Compose application model through multiple Compose files . When doing so, Compose follows the rules declared in this section to merge Compose files. Mapping A YAML mapping gets merged by adding missing entries and merging the conflicting ones. Merging the following example YAML trees: services: foo: key1: value1 key2: value2 services: foo: key2: VALUE key3: value3 Results in a Compose application model equivalent to the YAML tree: services: foo: key1: value1 key2: VALUE key3: value3 Sequence A YAML sequence is merged by appending values from the overriding Compose file to the previous one. Merging the following example YAML trees: services: foo: DNS: - 1.1.1.1 services: foo: DNS: - 8.8.8.8 Results in a Compose application model equivalent to the YAML tree: services: foo: DNS: - 1.1.1.1 - 8.8.8.8 Exceptions Shell commands When merging Compose files that use the services attributes command , entrypoint and healthcheck: test , the value is overridden by the latest Compose file, and not appended. Merging the following example YAML trees: services: foo: command: [\"echo\", \"foo\"] services: foo: command: [\"echo\", \"bar\"] Results in a Compose application model equivalent to the YAML tree: services: foo: command: [\"echo\", \"bar\"] Unique resources Applies to the ports , volumes , secrets and configs services attributes. While these types are modeled in a Compose file as a sequence, they have special uniqueness requirements: Attribute Unique key volumes target secrets source configs source ports {ip, target, published, protocol} When merging Compose files, Compose appends new entries that do not violate a uniqueness constraint and merge entries that share a unique key. Merging the following example YAML trees: services: foo: volumes: - foo:/work services: foo: volumes: - bar:/work Results in a Compose application model equivalent to the YAML tree: services: foo: volumes: - bar:/work Reset value In addition to the previously described mechanism, an override Compose file can also be used to remove elements from your application model. For this purpose, the custom YAML tag !reset can be set to override a value set by the overriden Compose file. A valid value for attribute must be provided, but will be ignored and target attribute will be set with type's default value or null . For readability, it is recommended to explicitly set the attribute value to the null ( null ) or empty array [] (with !reset null or !reset [] ) so that it is clear that resulting attribute will be cleared. A base compose.yaml file: services: app: image: myapp ports: - \"8080:80\" environment: FOO: BAR And an overide.compose.yaml file: services: app: image: myapp ports: !reset [] environment: FOO: !reset null Results in: services: app: image: myapp Replace value While !reset can be used to remove a declaration from a Compose file using an override file, !override allows you to fully replace an attribute, bypassing the standard merge rules. A typical example is to fully replace a resource definition, to rely on a distinct model but using the same name. A base compose.yaml file: services: app: image: myapp ports: - \"8080:80\" To remove the original port, but expose a new one, the following override file is used: services: app: ports: !override - \"8443:443\" This results in: services: app: image: myapp ports: - \"8443:443\" If !override had not been used, both 8080:80 and 8443:443 would be exposed as per the merging rules outlined above . Include A Compose application can declare dependency on another Compose application. This is useful if: - You want to reuse other Compose files. - You need to factor out parts of your application model into separate Compose files so they can be managed separately or shared with others. - Teams need to keep a Compose file reasonably complicated for the limited amount of resources it has to declare for it's own sub-domain, within a larger deployment. The include top-level section is used to define the dependency on another Compose application, or sub-domain. Each path listed in the include section is loaded as an individual Compose application model, with it's own project directory, in order to resolve relative paths. Once the included Compose application is loaded, all resources definitions are copied into the current Compose application model. Compose displays a warning if resource names conflict and doesn't try to merge them. To enforce this, include is evaluated after the Compose file(s) selected to define the Compose application model have been parsed and merged, so that conflicts between Compose files are detected. include applies recursively so an included Compose file which declares its own include section, triggers those other files to be included as well. Any volumes, networks, or other resources pulled in from the included Compose file can be used by the current Compose application for cross-service references. For example: include: - my-compose-include.yaml #with serviceB declared services: serviceA: build: . depends_on: - serviceB #use serviceB directly as if it was declared in this Compose file Compose also supports the use of interpolated variables with include . It's recommended that you specify mandatory variables . For example: include: -${INCLUDE_PATH:?FOO}/compose.yaml Short syntax The short syntax only defines paths to other Compose files. The file is loaded with the parent folder as the project directory, and an optional .env file that is loaded to define any variables' default values by interpolation. The local project's environment can override those values. include: - ../commons/compose.yaml - ../another_domain/compose.yaml services: webapp: depends_on: - included-service ## defined by another_domain In the above example, both ../commons/compose.yaml and ../another_domain/compose.yaml are loaded as individual Compose projects. Relative paths in Compose files being referred by include are resolved relative to their own Compose file path, not based on the local project's directory. Variables are interpolated using values set in the optional .env file in same folder, and is overridden by the local project's environment. Long syntax The long syntax offers more control over the sub-project parsing: include: - path: ../commons/compose.yaml project_directory: .. env_file: ../another/.env path path is required and defines the location of the Compose file(s) to be parsed and included into the local Compose model. path can be set either to a string when a single Compose file is involved, or to a list of strings when multiple Compose files need to be merged together to define the Compose model to be included in the local application. include: - path: - ../commons/compose.yaml - ./commons-override.yaml project_directory project_directory defines a base path to resolve relative paths set in the Compose file. It defaults to the directory of the included Compose file. env_file env_file defines an environment file(s) to use to define default values when interpolating variables in the Compose file being parsed. It defaults to .env file in the project_directory for the Compose file being parsed. env_file can be set either to a string or a list of strings when multiple environment files need to be merged to define a project environment. include: - path: ../another/compose.yaml env_file: - ../another/.env - ../another/dev.env The local project's environment has precedence over the values set by the Compose file, so that the local project can override values for customization. Profiles With profiles you can define a set of active profiles so your Compose application model is adjusted for various usages and environments. The exact mechanism is implementation specific and may include command line flags, environment variables, etc. The services top-level element supports a profiles attribute to define a list of named profiles. Services without a profiles attribute are always enabled. A service is ignored by Compose when none of the listed profiles match the active ones, unless the service is explicitly targeted by a command. In that case its profile is added to the set of active profiles. Note All other top-level elements are not affected by profiles and are always active. References to other services (by links , extends or shared resource syntax service:xxx ) do not automatically enable a component that would otherwise have been ignored by active profiles. Instead Compose returns an error. Illustrative example services: foo: image: foo bar: image: bar profiles: - test baz: image: baz depends_on: - bar profiles: - test zot: image: zot depends_on: - bar profiles: - debug In the above example: If the Compose application model is parsed with no profile enabled, it only contains the foo service. If the profile test is enabled, the model contains the services bar and baz , and service foo , which is always enabled. If the profile debug is enabled, the model contains both foo and zot services, but not bar and baz , and as such the model is invalid regarding the depends_on constraint of zot . If the profiles debug and test are enabled, the model contains all services; foo , bar , baz and zot . If Compose is executed with bar as the explicit service to run, bar and the test profile are active even if test profile is not enabled. If Compose is executed with baz as the explicit service to run, the service baz and the profile test are active and bar is pulled in by the depends_on constraint. If Compose is executed with zot as the explicit service to run, again the model is invalid regarding the depends_on constraint of zot , since zot and bar have no common profiles listed. If Compose is executed with zot as the explicit service to run and profile test is enabled, profile debug is automatically enabled and service bar is pulled in as a dependency starting both services zot and bar . See how you can use profiles in Docker Compose .","title":"Docker"},{"location":"docker/#docker","text":"sudo apt update && sudo apt upgrade -y sudo apt install git curl htop mc rsync zsh-autosuggestions sudo apt install zsh chsh -s $(which zsh) sh -c \"$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)\" rsync -avuPz ./.oh-my-zsh/custom/themes/custom.zsh-theme furyhawk@arm:/home/furyhawk/.oh-my-zsh/custom/themes/custom.zsh-theme git clone https://github.com/zsh-users/zsh-autosuggestions.git $ZSH_CUSTOM/plugins/zsh-autosuggestions git clone https://github.com/zdharma-continuum/fast-syntax-highlighting.git ${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/plugins/fast-syntax-highlighting ## Enable plugins by adding them to .zshrc. - Open .zshrc `nano ~/.zshrc` - ZSH_THEME=\"custom\" - Find the line which says `plugins=(git)`. - Replace that line with `plugins=(git zsh-autosuggestions zsh-syntax-highlighting fast-syntax-highlighting zsh-autocomplete)` touch .zprofile nano .zprofile # set PATH so it includes user's private bin if it exists if [ -d \"$HOME/bin\" ] ; then PATH=\"$HOME/bin:$PATH\" fi # set PATH so it includes user's private bin if it exists if [ -d \"$HOME/.local/bin\" ] ; then PATH=\"$HOME/.local/bin:$PATH\" fi export TZ=Asia/Singapore export NODE_ID=$(docker info -f '{{.Swarm.NodeID}}') export EMAIL=furyx@hotmail.com export DOMAIN=furyhawk.lol sudo nano /etc/fstab touch .credentials nano .credentials sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc sudo chmod a+r /etc/apt/keyrings/docker.asc echo \\ \"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \\ $(. /etc/os-release && echo \"$VERSION_CODENAME\") stable\" | \\ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin # sudo groupadd docker sudo usermod -aG docker $USER sudo systemctl enable docker.service sudo systemctl enable containerd.service systemctl list-units --type=service --state=active cd /etc/docker sudo touch daemon.json sudo nano daemon.json { \"log-driver\": \"json-file\", \"log-opts\": { \"max-size\": \"10m\", \"max-file\": \"3\" } } docker swarm join --token SWMTKN-1-xxx 192.168.50.114:2377 docker node update --availability drain node docker node update --availability Active node curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm chmod +x cephadm sudo ./cephadm add-repo --release octopus sudo ./cephadm install cephadm install ceph-common mkdir -p /etc/ceph sudo ./cephadm bootstrap --mon-ip 192.168.65.19 URL: https://debian.local:8443/ User: admin Password: 12345678 You can access the Ceph CLI with: sudo ./cephadm shell --fsid 8ad7deb6-265a-11ef-9e81-c2d0c41fc7e0 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring Please consider enabling telemetry to help improve Ceph: ceph telemetry on For more information see: https://docs.ceph.com/docs/master/mgr/telemetry/ alias ceph='./cephadm shell -- ceph' sudo ceph -v ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) sudo ceph orch host label add debian mon Added label mon to host debian sudo ceph orch apply mon debian Scheduled mon update... sudo ceph orch host ls ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-2 ssh-copy-id -f -i /etc/ceph/ceph.pub osd1 ssh-copy-id -f -i /etc/ceph/ceph.pub osd2 ssh-copy-id -f -i /etc/ceph/ceph.pub osd3 sudo ceph orch host add osd1 sudo ceph orch host add osd2 sudo ceph orch host add osd3 sudo ceph orch apply osd --all-available-devices sudo ceph status sudo find / -iname 'cephadm*' 2>/dev/null systemctl --user start docker-desktop sudo groupadd docker sudo usermod -aG docker $USER groups ${USER} curl -fsSL test.docker.com -o get-docker.sh && sh get-docker.sh sudo apt-get install libffi-dev libssl-dev sudo apt install python3-dev sudo apt-get install -y python3 python3-pip pip install \"cython<3.0.0\" wheel pip install \"pyyaml==5.4.1\" --no-build-isolation pip install docker-compose sudo systemctl enable docker sudo apt-get purge docker-ce sudo apt-get purge docker-ce-cli sudo rm -rf /var/lib/docker rootless sudo apt install -y dbus-user-session sudo apt install -y fuse-overlayfs sudo apt install -y fuse-overlayfs sudo systemctl disable --now docker.service docker.socket sudo rm -rf /var/lib/docker sudo apt-get purge docker-ce docker-ce-cli containerd.io sudo apt-get purge docker-ce docker-ce-cli containerd.io apt-get install -y uidmap curl -fsSL https://get.docker.com/rootless | sh nano .zshrc export PATH=/home/furyhawk/bin:$PATH export DOCKER_HOST=unix:///run/user/1000/docker.sock systemctl --user enable docker sudo loginctl enable-linger $(whoami) docker context use rootless sudo setcap cap_net_bind_service=ep $(which rootlesskit) systemctl --user restart docker #sudo apt-get install -y docker-ce-rootless-extras dockerd-rootless-setuptool.sh uninstall cd ~/bin rm -f containerd containerd-shim containerd-shim-runc-v2 ctr docker docker-init docker-proxy dockerd dockerd-rootless-setuptool.sh dockerd-rootless.sh rootlesskit rootlesskit-docker-proxy runc vpnkit netstat -ltup","title":"Docker"},{"location":"docker/#the-compose-specification","text":"{:.no_toc} ToC {:toc}","title":"The Compose Specification"},{"location":"docker/#status-of-this-document","text":"This document specifies the Compose file format used to define multi-containers applications. Distribution of this document is unlimited.","title":"Status of this document"},{"location":"docker/#requirements-and-optional-attributes","text":"The Compose specification includes properties designed to target a local OCI container runtime, exposing Linux kernel specific configuration options, but also some Windows container specific properties. It is also designed for cloud platform features related to resource placement on a cluster, replicated application distribution, and scalability. We acknowledge that no Compose implementation is expected to support all attributes, and that support for some properties is platform dependent and can only be confirmed at runtime. The definition of a versioned schema to control the supported properties in a Compose file, established by the docker-compose tool where the Compose file format was designed, doesn't offer any guarantee to the end-user that attributes will be actually implemented. The specification defines the expected configuration syntax and behavior. Unless noted, supporting any of these is optional. A Compose implementation to parse a Compose file using unsupported attributes should warn users. We recommend the following implementors to support those running modes: Default: warn the user about unsupported attributes, but ignore them Strict: warn the user about unsupported attributes and reject the Compose file Loose: ignore unsupported attributes AND unknown attributes (that were not defined by the spec by the time implementation was created) From this point onwards, references made to 'Compose' can be interpreted as 'a Compose implementation'.","title":"Requirements and optional attributes"},{"location":"docker/#the-compose-application-model","text":"The Compose Specification lets you define a platform-agnostic container based application. Such an application is designed as a set of containers which have to both run together with adequate shared resources and communication channels. Computing components of an application are defined as services . A service is an abstract concept implemented on platforms by running the same container image, and configuration, one or more times. Services communicate with each other through networks . In the Compose Specification, a network is a platform capability abstraction to establish an IP route between containers within services connected together. Low-level, platform-specific networking options are grouped into the Network definition and may be partially implemented on some platforms. Services store and share persistent data into volumes . The Specification describes such a persistent data as a high-level filesystem mount with global options. Actual platform-specific implementation details are grouped into the volumes definition and may be partially implemented on some platforms. Some services require configuration data that is dependent on the runtime or platform. For this, the Specification defines a dedicated configs concept. From a service container point of view, configs are comparable to volumes, in that they are files mounted into the container. But the actual definition involves distinct platform resources and services, which are abstracted by this type. A secret is a specific flavor of configuration data for sensitive data that should not be exposed without security considerations. Secrets are made available to services as files mounted into their containers, but the platform-specific resources to provide sensitive data are specific enough to deserve a distinct concept and definition within the Compose specification. Note With volumes, configs and secrets you can have a simple declaration at the top-level and then add more platform-specific information at the service level. A project is an individual deployment of an application specification on a platform. A project's name, set with the top-level name attribute, is used to group resources together and isolate them from other applications or other installation of the same Compose specified application with distinct parameters. If you are creating resources on a platform, you must prefix resource names by project and set the label com.docker.compose.project . Compose offers a way for users to set a custom project name and override this name, so that the same compose.yaml file can be deployed twice on the same infrastructure, without changes, by just passing a distinct name. Project names must contain only lowercase letters, decimal digits, dashes, and underscores, and must begin with a lowercase letter or decimal digit.","title":"The Compose application model"},{"location":"docker/#illustrative-example","text":"The following example illustrates the Compose Specification concepts outlined above. The example is non-normative. Consider an application split into a frontend web application and a backend service. The frontend is configured at runtime with an HTTP configuration file managed by infrastructure, providing an external domain name, and an HTTPS server certificate injected by the platform's secured secret store. The backend stores data in a persistent volume. Both services communicate with each other on an isolated back-tier network, while the frontend is also connected to a front-tier network and exposes port 443 for external usage. %%{ init: { 'flowchart': { 'curve': 'linear' } } }%% flowchart LR subgraph A[INFRASTRUCTURE] direction TB subgraph TOP[\" \"] subgraph B1[Frontend Service] fs[\"`**webapp**`\"] end style B1 fill:#ccd6e8, stroke-width:0px subgraph B2[Backend Service] bs[\"`**database**`\"] end style B2 fill:#ccd6e8, stroke-width:0px end style TOP fill:transparent, stroke-width:2px, stroke:#62affb, stroke-dasharray: 5 5 key[ro= read only\\nr+w = read write] style key fill:transparent, stroke-width:0px,text-align: left, size: 94px direction TB id2(Server\\nCertificate) id1(HTTP\\nConfiguration) id1 & id2 -.-|ro| B1 style id1 stroke:#000,stroke-width:1px,stroke-dasharray: 10 style id2 stroke:#000,stroke-width:1px,stroke-dasharray: 10 B2 ==r+w==> id3[(Persistent\\nVolume)] end style A fill:#eeeeee, stroke-width:0px direction LR id4[External\\nUser] ---id5(((443)))--->|Frontend\\nNetwork| B1 style id4 stroke:#000,stroke-width:2px B1 --Backend\\nNetwork--> B2 The example application is composed of the following parts: 2 services, backed by Docker images: webapp and database 1 secret (HTTPS certificate), injected into the frontend 1 configuration (HTTP), injected into the frontend 1 persistent volume, attached to the backend 2 networks services: frontend: image: example/webapp ports: - \"443:8043\" networks: - front-tier - back-tier configs: - httpd-config secrets: - server-certificate backend: image: example/database volumes: - db-data:/etc/data networks: - back-tier volumes: db-data: driver: flocker driver_opts: size: \"10GiB\" configs: httpd-config: external: true secrets: server-certificate: external: true networks: ## The presence of these objects is sufficient to define them front-tier: {} back-tier: {} This example illustrates the distinction between volumes, configs and secrets. While all of them are all exposed to service containers as mounted files or directories, only a volume can be configured for read+write access. Secrets and configs are read-only. The volume configuration allows you to select a volume driver and pass driver options to tweak volume management according to the actual infrastructure. Configs and secrets rely on platform services, and are declared external as they are not managed as part of the application lifecycle. Compose uses a platform-specific lookup mechanism to retrieve runtime values.","title":"Illustrative example"},{"location":"docker/#compose-file","text":"The Compose file is a YAML file defining: - Version (Optional) - Services (Required) - Networks - Volumes - Configs - Secrets The default path for a Compose file is compose.yaml (preferred) or compose.yml that is placed in the working directory. Compose also supports docker-compose.yaml and docker-compose.yml for backwards compatibility of earlier versions. If both files exist, Compose prefers the canonical compose.yaml . You can use fragments and extensions to keep your Compose file efficient and easy to maintain. Multiple Compose files can be merged together to define the application model. The combination of YAML files are implemented by appending or overriding YAML elements based on the Compose file order you set. Simple attributes and maps get overridden by the highest order Compose file, lists get merged by appending. Relative paths are resolved based on the first Compose file's parent folder, whenever complimentary files being merged are hosted in other folders. As some Compose file elements can both be expressed as single strings or complex objects, merges apply to the expanded form. If you want to reuse other Compose files, or factor out parts of you application model into separate Compose files, you can also use include . This is useful if your Compose application is dependent on another application which is managed by a different team, or needs to be shared with others.","title":"Compose file"},{"location":"docker/#version-and-name-top-level-elements","text":"","title":"Version and name top-level elements"},{"location":"docker/#version-top-level-element","text":"The top-level version property is defined by the Compose Specification for backward compatibility. It is only informative. Compose doesn't use version to select an exact schema to validate the Compose file, but prefers the most recent schema when it's implemented. Compose validates whether it can fully parse the Compose file. If some fields are unknown, typically because the Compose file was written with fields defined by a newer version of the Specification, you'll receive a warning message. Compose offers options to ignore unknown fields (as defined by \"loose\" mode).","title":"Version top-level element"},{"location":"docker/#name-top-level-element","text":"The top-level name property is defined by the Specification as the project name to be used if you don't set one explicitly. Compose offers a way for you to override this name, and sets a default project name to be used if the top-level name element is not set. Whenever a project name is defined by top-level name or by some custom mechanism, it is exposed for interpolation and environment variable resolution as COMPOSE_PROJECT_NAME services: foo: image: busybox environment: - COMPOSE_PROJECT_NAME command: echo \"I'm running ${COMPOSE_PROJECT_NAME}\"","title":"Name top-level element"},{"location":"docker/#services-top-level-element","text":"A service is an abstract definition of a computing resource within an application which can be scaled or replaced independently from other components. Services are backed by a set of containers, run by the platform according to replication requirements and placement constraints. As services are backed by containers, they are defined by a Docker image and set of runtime arguments. All containers within a service are identically created with these arguments. A Compose file must declare a services top-level element as a map whose keys are string representations of service names, and whose values are service definitions. A service definition contains the configuration that is applied to each service container. Each service may also include a build section, which defines how to create the Docker image for the service. Compose supports building docker images using this service definition. If not used, the build section is ignored and the Compose file is still considered valid. Build support is an optional aspect of the Compose Specification, and is described in detail in the Compose Build Specification documentation. Each service defines runtime constraints and requirements to run its containers. The deploy section groups these constraints and allows the platform to adjust the deployment strategy to best match containers' needs with available resources. Deploy support is an optional aspect of the Compose Specification, and is described in detail in the Compose Deploy Specification documentation. If not implemented the deploy section is ignored and the Compose file is still considered valid.","title":"Services top-level element"},{"location":"docker/#attach","text":"When attach is defined and set to false Compose does not collect service logs, until you explicitly request it to. The default service configuration is attach: true .","title":"attach"},{"location":"docker/#build","text":"build specifies the build configuration for creating a container image from source, as defined in the Compose Build Specification .","title":"build"},{"location":"docker/#blkio_config","text":"blkio_config defines a set of configuration options to set block IO limits for a service. services: foo: image: busybox blkio_config: weight: 300 weight_device: - path: /dev/sda weight: 400 device_read_bps: - path: /dev/sdb rate: '12mb' device_read_iops: - path: /dev/sdb rate: 120 device_write_bps: - path: /dev/sdb rate: '1024k' device_write_iops: - path: /dev/sdb rate: 30","title":"blkio_config"},{"location":"docker/#device_read_bps-device_write_bps","text":"Set a limit in bytes per second for read / write operations on a given device. Each item in the list must have two keys: path : Defines the symbolic path to the affected device. rate : Either as an integer value representing the number of bytes or as a string expressing a byte value.","title":"device_read_bps, device_write_bps"},{"location":"docker/#device_read_iops-device_write_iops","text":"Set a limit in operations per second for read / write operations on a given device. Each item in the list must have two keys: path : Defines the symbolic path to the affected device. rate : As an integer value representing the permitted number of operations per second.","title":"device_read_iops, device_write_iops"},{"location":"docker/#weight","text":"Modify the proportion of bandwidth allocated to a service relative to other services. Takes an integer value between 10 and 1000, with 500 being the default.","title":"weight"},{"location":"docker/#weight_device","text":"Fine-tune bandwidth allocation by device. Each item in the list must have two keys: path : Defines the symbolic path to the affected device. weight : An integer value between 10 and 1000.","title":"weight_device"},{"location":"docker/#cpu_count","text":"cpu_count defines the number of usable CPUs for service container.","title":"cpu_count"},{"location":"docker/#cpu_percent","text":"cpu_percent defines the usable percentage of the available CPUs.","title":"cpu_percent"},{"location":"docker/#cpu_shares","text":"cpu_shares defines, as integer value, a service container's relative CPU weight versus other containers.","title":"cpu_shares"},{"location":"docker/#cpu_period","text":"cpu_period configures CPU CFS (Completely Fair Scheduler) period when a platform is based on Linux kernel.","title":"cpu_period"},{"location":"docker/#cpu_quota","text":"cpu_quota configures CPU CFS (Completely Fair Scheduler) quota when a platform is based on Linux kernel.","title":"cpu_quota"},{"location":"docker/#cpu_rt_runtime","text":"cpu_rt_runtime configures CPU allocation parameters for platforms with support for realtime scheduler. It can be either an integer value using microseconds as unit or a duration . cpu_rt_runtime: '400ms' cpu_rt_runtime: 95000`","title":"cpu_rt_runtime"},{"location":"docker/#cpu_rt_period","text":"cpu_rt_period configures CPU allocation parameters for platforms with support for realtime scheduler. It can be either an integer value using microseconds as unit or a duration . cpu_rt_period: '1400us' cpu_rt_period: 11000`","title":"cpu_rt_period"},{"location":"docker/#cpus","text":"DEPRECATED: use deploy.limits.cpus cpus define the number of (potentially virtual) CPUs to allocate to service containers. This is a fractional number. 0.000 means no limit.","title":"cpus"},{"location":"docker/#cpuset","text":"cpuset defines the explicit CPUs in which to allow execution. Can be a range 0-3 or a list 0,1","title":"cpuset"},{"location":"docker/#cap_add","text":"cap_add specifies additional container capabilities as strings. cap_add: - ALL","title":"cap_add"},{"location":"docker/#cap_drop","text":"cap_drop specifies container capabilities to drop as strings. cap_drop: - NET_ADMIN - SYS_ADMIN","title":"cap_drop"},{"location":"docker/#cgroup","text":"cgroup specifies the cgroup namespace to join. When unset, it is the container runtime's decision to select which cgroup namespace to use, if supported. host : Runs the container in the Container runtime cgroup namespace. private : Runs the container in its own private cgroup namespace.","title":"cgroup"},{"location":"docker/#cgroup_parent","text":"cgroup_parent specifies an optional parent cgroup for the container. cgroup_parent: m-executor-abcd","title":"cgroup_parent"},{"location":"docker/#command","text":"command overrides the default command declared by the container image, for example by Dockerfile's CMD . command: bundle exec thin -p 3000 The value can also be a list, in a manner similar to Dockerfile : command: [ \"bundle\", \"exec\", \"thin\", \"-p\", \"3000\" ] If the value is null , the default command from the image is used. If the value is [] (empty list) or '' (empty string), the default command declared by the image is ignored, i.e. overridden to be empty.","title":"command"},{"location":"docker/#configs","text":"Configs allow services to adapt their behaviour without the need to rebuild a Docker image. Services can only access configs when explicitly granted by the configs attribute. Two different syntax variants are supported. Compose reports an error if config doesn't exist on the platform or isn't defined in the configs top-level element in the Compose file. There are two syntaxes defined for configs. To remain compliant to this specification, an implementation must support both syntaxes. Implementations must allow use of both short and long syntaxes within the same document. You can grant a service access to multiple configs, and you can mix long and short syntax.","title":"configs"},{"location":"docker/#short-syntax","text":"The short syntax variant only specifies the config name. This grants the container access to the config and mounts it as files into a service\u2019s container\u2019s filesystem. The location of the mount point within the container defaults to / in Linux containers, and C:\\ in Windows containers. The following example uses the short syntax to grant the redis service access to the my_config and my_other_config configs. The value of my_config is set to the contents of the file ./my_config.txt , and my_other_config is defined as an external resource, which means that it has already been defined in the platform. If the external config does not exist, the deployment fails. services: redis: image: redis:latest configs: - my_config - my_other_config configs: my_config: file: ./my_config.txt my_other_config: external: true","title":"Short syntax"},{"location":"docker/#long-syntax","text":"The long syntax provides more granularity in how the config is created within the service's task containers. source : The name of the config as it exists in the platform. target : The path and name of the file to be mounted in the service's task containers. Defaults to / if not specified. uid and gid : The numeric UID or GID that owns the mounted config file within the service's task containers. Default value when not specified is USER running container. mode : The permissions for the file that is mounted within the service's task containers, in octal notation. Default value is world-readable ( 0444 ). Writable bit must be ignored. The executable bit can be set. The following example sets the name of my_config to redis_config within the container, sets the mode to 0440 (group-readable) and sets the user and group to 103 . The redis service does not have access to the my_other_config config. services: redis: image: redis:latest configs: - source: my_config target: /redis_config uid: \"103\" gid: \"103\" mode: 0440 configs: my_config: external: true my_other_config: external: true","title":"Long syntax"},{"location":"docker/#container_name","text":"container_name is a string that specifies a custom container name, rather than a name generated by default. container_name: my-web-container Compose does not scale a service beyond one container if the Compose file specifies a container_name . Attempting to do so results in an error. container_name follows the regex format of [a-zA-Z0-9][a-zA-Z0-9_.-]+","title":"container_name"},{"location":"docker/#credential_spec","text":"credential_spec configures the credential spec for a managed service account. If you have services that use Windows containers, you can use file: and registry: protocols for credential_spec . Compose also supports additional protocols for custom use-cases. The credential_spec must be in the format file:// or registry:// . credential_spec: file: my-credential-spec.json When using registry: , the credential spec is read from the Windows registry on the daemon's host. A registry value with the given name must be located in: HKLM\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Virtualization\\Containers\\CredentialSpecs The following example loads the credential spec from a value named my-credential-spec in the registry: credential_spec: registry: my-credential-spec","title":"credential_spec"},{"location":"docker/#example-gmsa-configuration","text":"When configuring a gMSA credential spec for a service, you only need to specify a credential spec with config , as shown in the following example: services: myservice: image: myimage:latest credential_spec: config: my_credential_spec configs: my_credentials_spec: file: ./my-credential-spec.json|","title":"Example gMSA configuration"},{"location":"docker/#depends_on","text":"depends_on expresses startup and shutdown dependencies between services.","title":"depends_on"},{"location":"docker/#short-syntax_1","text":"The short syntax variant only specifies service names of the dependencies. Service dependencies cause the following behaviors: Compose creates services in dependency order. In the following example, db and redis are created before web . Compose removes services in dependency order. In the following example, web is removed before db and redis . Simple example: services: web: build: . depends_on: - db - redis redis: image: redis db: image: postgres Compose guarantees dependency services have been started before starting a dependent service. Compose waits for dependency services to be \"ready\" before starting a dependent service.","title":"Short syntax"},{"location":"docker/#long-syntax_1","text":"The long form syntax enables the configuration of additional fields that can't be expressed in the short form. restart : When set to true Compose restarts this service after it updates the dependency service. This applies to an explicit restart controlled by a Compose operation, and excludes automated restart by the container runtime after the container dies. condition : Sets the condition under which dependency is considered satisfied service_started : An equivalent of the short syntax described above service_healthy : Specifies that a dependency is expected to be \"healthy\" (as indicated by healthcheck ) before starting a dependent service. service_completed_successfully : Specifies that a dependency is expected to run to successful completion before starting a dependent service. required : When set to false Compose only warns you when the dependency service isn't started or available. If it's not defined the default value of required is true . Service dependencies cause the following behaviors: Compose creates services in dependency order. In the following example, db and redis are created before web . Compose waits for healthchecks to pass on dependencies marked with service_healthy . In the following example, db is expected to be \"healthy\" before web is created. Compose removes services in dependency order. In the following example, web is removed before db and redis . services: web: build: . depends_on: db: condition: service_healthy restart: true redis: condition: service_started redis: image: redis db: image: postgres Compose guarantees dependency services are started before starting a dependent service. Compose guarantees dependency services marked with service_healthy are \"healthy\" before starting a dependent service.","title":"Long syntax"},{"location":"docker/#deploy","text":"deploy specifies the configuration for the deployment and lifecycle of services, as defined in the Compose Deploy Specification .","title":"deploy"},{"location":"docker/#develop","text":"develop specifies the development configuration for maintaining a container in sync with source, as defined in the Development Section .","title":"develop"},{"location":"docker/#device_cgroup_rules","text":"device_cgroup_rules defines a list of device cgroup rules for this container. The format is the same format the Linux kernel specifies in the Control Groups Device Whitelist Controller . device_cgroup_rules: - 'c 1:3 mr' - 'a 7:* rmw'","title":"device_cgroup_rules"},{"location":"docker/#devices","text":"devices defines a list of device mappings for created containers in the form of HOST_PATH:CONTAINER_PATH[:CGROUP_PERMISSIONS] . devices: - \"/dev/ttyUSB0:/dev/ttyUSB0\" - \"/dev/sda:/dev/xvda:rwm\"","title":"devices"},{"location":"docker/#dns","text":"dns defines custom DNS servers to set on the container network interface configuration. It can be a single value or a list. dns: 8.8.8.8 dns: - 8.8.8.8 - 9.9.9.9","title":"dns"},{"location":"docker/#dns_opt","text":"dns_opt list custom DNS options to be passed to the container\u2019s DNS resolver ( /etc/resolv.conf file on Linux). dns_opt: - use-vc - no-tld-query","title":"dns_opt"},{"location":"docker/#dns_search","text":"dns_search defines custom DNS search domains to set on container network interface configuration. It can be a single value or a list. dns_search: example.com dns_search: - dc1.example.com - dc2.example.com","title":"dns_search"},{"location":"docker/#domainname","text":"domainname declares a custom domain name to use for the service container. It must be a valid RFC 1123 hostname.","title":"domainname"},{"location":"docker/#entrypoint","text":"entrypoint declares the default entrypoint for the service container. This overrides the ENTRYPOINT instruction from the service's Dockerfile. If entrypoint is non-null, Compose ignores any default command from the image, for example the CMD instruction in the Dockerfile. See also command to set or override the default command to be executed by the entrypoint process. In its short form, the value can be defined as a string: entrypoint: /code/entrypoint.sh Alternatively, the value can also be a list, in a manner similar to the Dockerfile : entrypoint: - php - -d - zend_extension=/usr/local/lib/php/extensions/no-debug-non-zts-20100525/xdebug.so - -d - memory_limit=-1 - vendor/bin/phpunit If the value is null , the default entrypoint from the image is used. If the value is [] (empty list) or '' (empty string), the default entrypoint declared by the image is ignored, i.e. overridden to be empty.","title":"entrypoint"},{"location":"docker/#env_file","text":"env_file adds environment variables to the container based on the file content. env_file: .env env_file can also be a list. The files in the list are processed from the top down. For the same variable specified in two env files, the value from the last file in the list stands. env_file: - ./a.env - ./b.env List elements can also be declared as a mapping, which then lets you set an additional attribute required . This defaults to true . When required is set to false and the .env file is missing, Compose silently ignores the entry. env_file: - path: ./default.env required: true ## default - path: ./override.env required: false Relative path are resolved from the Compose file's parent folder. As absolute paths prevent the Compose file from being portable, Compose warns you when such a path is used to set env_file . Environment variables declared in the environment section override these values. This holds true even if those values are empty or undefined.","title":"env_file"},{"location":"docker/#env_file-format","text":"Each line in an .env file must be in VAR[=[VAL]] format. The following syntax rules apply: Lines beginning with # are processed as comments and ignored. Blank lines are ignored. Unquoted and double-quoted ( \" ) values have Interpolation applied. Each line represents a key-value pair. Values can optionally be quoted. VAR=VAL -> VAL VAR=\"VAL\" -> VAL VAR='VAL' -> VAL Inline comments for unquoted values must be preceded with a space. VAR=VAL ## comment -> VAL VAR=VAL## not a comment -> VAL## not a comment Inline comments for quoted values must follow the closing quote. VAR=\"VAL ## not a comment\" -> VAL ## not a comment VAR=\"VAL\" ## comment -> VAL Single-quoted ( ' ) values are used literally. VAR='$OTHER' -> $OTHER VAR='${OTHER}' -> ${OTHER} Quotes can be escaped with \\ . VAR='Let\\'s go!' -> Let's go! VAR=\"{\\\"hello\\\": \\\"json\\\"}\" -> {\"hello\": \"json\"} Common shell escape sequences including \\n , \\r , \\t , and \\\\ are supported in double-quoted values. VAR=\"some\\tvalue\" -> some value VAR='some\\tvalue' -> some\\tvalue VAR=some\\tvalue -> some\\tvalue VAL may be omitted, in such cases the variable value is an empty string. =VAL may be omitted, in such cases the variable is unset. ## Set Rails/Rack environment RACK_ENV=development VAR=\"quoted\"","title":"Env_file format"},{"location":"docker/#environment","text":"environment defines environment variables set in the container. environment can use either an array or a map. Any boolean values; true, false, yes, no, should be enclosed in quotes to ensure they are not converted to True or False by the YAML parser. Environment variables can be declared by a single key (no value to equals sign). In this case Compose relies on you to resolve the value. If the value is not resolved, the variable is unset and is removed from the service container environment. Map syntax: environment: RACK_ENV: development SHOW: \"true\" USER_INPUT: Array syntax: environment: - RACK_ENV=development - SHOW=true - USER_INPUT When both env_file and environment are set for a service, values set by environment have precedence.","title":"environment"},{"location":"docker/#expose","text":"expose defines the (incoming) port or a range of ports that Compose exposes from the container. These ports must be accessible to linked services and should not be published to the host machine. Only the internal container ports can be specified. Syntax is /[] or /[] for a port range. When not explicitly set, tcp protocol is used. expose: - \"3000\" - \"8000\" - \"8080-8085/tcp Note If the Dockerfile for the image already exposes ports, it is visible to other containers on the network even if expose is not set in your Compose file.","title":"expose"},{"location":"docker/#extends","text":"extends lets you share common configurations among different files, or even different projects entirely. With extends you can define a common set of service options in one place and refer to it from anywhere. You can refer to another Compose file and select a service you want to also use in your own application, with the ability to override some attributes for your own needs. You can use extends on any service together with other configuration keys. The extends value must be a mapping defined with a required service and an optional file key. extends: file: common.yml service: webapp service : Defines the name of the service being referenced as a base, for example web or database . file : The location of a Compose configuration file defining that service.","title":"extends"},{"location":"docker/#restrictions","text":"The following restrictions apply to the service being referenced: Services that have dependencies on other services cannot be used as a base. Therefore, any key that introduces a dependency on another service is incompatible with extends . The non-exhaustive list of such keys is: links , volumes_from , container mode (in ipc , pid , network_mode and net ), service mode (in ipc , pid and network_mode ), depends_on . Services cannot have circular references with extends . Compose returns an error in all of these cases.","title":"Restrictions"},{"location":"docker/#finding-referenced-service","text":"file value can be: Not present. This indicates that another service within the same Compose file is being referenced. File path, which can be either: Relative path. This path is considered as relative to the location of the main Compose file. Absolute path. A service denoted by service must be present in the identified referenced Compose file. Compose returns an error if: The service denoted by service is not found. The Compose file denoted by file is not found.","title":"Finding referenced service"},{"location":"docker/#merging-service-definitions","text":"Two service definitions, the main one in the current Compose file and the referenced one specified by extends , are merged in the following way: Mappings: Keys in mappings of the main service definition override keys in mappings of the referenced service definition. Keys that aren't overridden are included as is. Sequences: Items are combined together into a new sequence. The order of elements is preserved with the referenced items coming first and main items after. Scalars: Keys in the main service definition take precedence over keys in the referenced one.","title":"Merging service definitions"},{"location":"docker/#mappings","text":"The following keys should be treated as mappings: annotations , build.args , build.labels , build.extra_hosts , deploy.labels , deploy.update_config , deploy.rollback_config , deploy.restart_policy , deploy.resources.limits , environment , healthcheck , labels , logging.options , sysctls , storage_opt , extra_hosts , ulimits . One exception that applies to healthcheck is that the main mapping cannot specify disable: true unless the referenced mapping also specifies disable: true . Compose returns an error in this case. For example, the input below: services: common: image: busybox environment: TZ: utc PORT: 80 cli: extends: service: common environment: PORT: 8080 Produces the following configuration for the cli service. The same output is produced if array syntax is used. environment: PORT: 8080 TZ: utc image: busybox Items under blkio_config.device_read_bps , blkio_config.device_read_iops , blkio_config.device_write_bps , blkio_config.device_write_iops , devices and volumes are also treated as mappings where key is the target path inside the container. For example, the input below: services: common: image: busybox volumes: - common-volume:/var/lib/backup/data:rw cli: extends: service: common volumes: - cli-volume:/var/lib/backup/data:ro Produces the following configuration for the cli service. Note that the mounted path now points to the new volume name and ro flag was applied. image: busybox volumes: - cli-volume:/var/lib/backup/data:ro If the referenced service definition contains extends mapping, the items under it are simply copied into the new merged definition. The merging process is then kicked off again until no extends keys are remaining. For example, the input below: services: base: image: busybox user: root common: image: busybox extends: service: base cli: extends: service: common Produces the following configuration for the cli service. Here, cli services gets user key from common service, which in turn gets this key from base service. image: busybox user: root","title":"Mappings"},{"location":"docker/#sequences","text":"The following keys should be treated as sequences: cap_add , cap_drop , configs , deploy.placement.constraints , deploy.placement.preferences , deploy.reservations.generic_resources , device_cgroup_rules , expose , external_links , ports , secrets , security_opt . Any duplicates resulting from the merge are removed so that the sequence only contains unique elements. For example, the input below: services: common: image: busybox security_opt: - label:role:ROLE cli: extends: service: common security_opt: - label:user:USER Produces the following configuration for the cli service. image: busybox security_opt: - label:role:ROLE - label:user:USER In case list syntax is used, the following keys should also be treated as sequences: dns , dns_search , env_file , tmpfs . Unlike sequence fields mentioned above, duplicates resulting from the merge are not removed.","title":"Sequences"},{"location":"docker/#scalars","text":"Any other allowed keys in the service definition should be treated as scalars.","title":"Scalars"},{"location":"docker/#annotations","text":"annotations defines annotations for the container. annotations can use either an array or a map. annotations: com.example.foo: bar annotations: - com.example.foo=bar","title":"annotations"},{"location":"docker/#external_links","text":"external_links link service containers to services managed outside of your Compose application. external_links define the name of an existing service to retrieve using the platform lookup mechanism. An alias of the form SERVICE:ALIAS can be specified. external_links: - redis - database:mysql - database:postgresql","title":"external_links"},{"location":"docker/#extra_hosts","text":"extra_hosts adds hostname mappings to the container network interface configuration ( /etc/hosts for Linux).","title":"extra_hosts"},{"location":"docker/#short-syntax_2","text":"Short syntax uses plain strings in a list. Values must set hostname and IP address for additional hosts in the form of HOSTNAME=IP . extra_hosts: - \"somehost=162.242.195.82\" - \"otherhost=50.31.209.229\" - \"myhostv6=::1\" IPv6 addresses can be enclosed in square brackets, for example: extra_hosts: - \"myhostv6=[::1]\" The separator = is preferred, but : can also be used. For example: extra_hosts: - \"somehost:162.242.195.82\" - \"myhostv6:::1\"","title":"Short syntax"},{"location":"docker/#long-syntax_2","text":"Alternatively, extra_hosts can be set as a mapping between hostname(s) and IP(s) extra_hosts: somehost: \"162.242.195.82\" otherhost: \"50.31.209.229\" myhostv6: \"::1\" Compose creates a matching entry with the IP address and hostname in the container's network configuration, which means for Linux /etc/hosts get extra lines: 162.242.195.82 somehost 50.31.209.229 otherhost ::1 myhostv6","title":"Long syntax"},{"location":"docker/#group_add","text":"group_add specifies additional groups, by name or number, which the user inside the container must be a member of. An example of where this is useful is when multiple containers (running as different users) need to all read or write the same file on a shared volume. That file can be owned by a group shared by all the containers, and specified in group_add . services: myservice: image: alpine group_add: - mail Running id inside the created container must show that the user belongs to the mail group, which would not have been the case if group_add were not declared.","title":"group_add"},{"location":"docker/#healthcheck","text":"healthcheck declares a check that's run to determine whether or not the service containers are \"healthy\". It works in the same way, and has the same default values, as the HEALTHCHECK Dockerfile instruction set by the service's Docker image. Your Compose file can override the values set in the Dockerfile. healthcheck: test: [\"CMD\", \"curl\", \"-f\", \"http://localhost\"] interval: 1m30s timeout: 10s retries: 3 start_period: 40s start_interval: 5s interval , timeout , start_period , and start_interval are specified as durations . test defines the command Compose runs to check container health. It can be either a string or a list. If it's a list, the first item must be either NONE , CMD or CMD-SHELL . If it's a string, it's equivalent to specifying CMD-SHELL followed by that string. ## Hit the local web app test: [\"CMD\", \"curl\", \"-f\", \"http://localhost\"] Using CMD-SHELL runs the command configured as a string using the container's default shell ( /bin/sh for Linux). Both forms below are equivalent: test: [\"CMD-SHELL\", \"curl -f http://localhost || exit 1\"] test: curl -f https://localhost || exit 1 NONE disables the healthcheck, and is mostly useful to disable the Healthcheck Dockerfile instruction set by the service's Docker image. Alternatively, the healthcheck set by the image can be disabled by setting disable: true : healthcheck: disable: true","title":"healthcheck"},{"location":"docker/#hostname","text":"hostname declares a custom host name to use for the service container. It must be a valid RFC 1123 hostname.","title":"hostname"},{"location":"docker/#image","text":"image specifies the image to start the container from. image must follow the Open Container Specification addressable image format , as [/][/][:|@] . image: redis image: redis:5 image: redis@sha256:0ed5d5928d4737458944eb604cc8509e245c3e19d02ad83935398bc4b991aac7 image: library/redis image: docker.io/library/redis image: my_private.registry:5000/redis If the image does not exist on the platform, Compose attempts to pull it based on the pull_policy . If you are also using the Compose Build Specification , there are alternative options for controlling the precedence of pull over building the image from source, however pulling the image is the default behavior. image may be omitted from a Compose file as long as a build section is declared. If you are not using the Compose Build Specification, Compose won't work if image is missing from the Compose file.","title":"image"},{"location":"docker/#init","text":"init runs an init process (PID 1) inside the container that forwards signals and reaps processes. Set this option to true to enable this feature for the service. services: web: image: alpine:latest init: true The init binary that is used is platform specific.","title":"init"},{"location":"docker/#ipc","text":"ipc configures the IPC isolation mode set by the service container. Available values are platform specific, but Compose defines specific values which must be implemented as described if supported: shareable : Gives the container its own private IPC namespace, with a possibility to share it with other containers. service:{name} : Makes the container join another container's ( shareable ) IPC namespace. ipc: \"shareable\" ipc: \"service:[service name]\"","title":"ipc"},{"location":"docker/#uts","text":"uts configures the UTS namespace mode set for the service container. When unspecified it is the runtime's decision to assign a UTS namespace, if supported. Available values are: 'host' : Results in the container using the same UTS namespace as the host. uts: \"host\"","title":"uts"},{"location":"docker/#isolation","text":"isolation specifies a container\u2019s isolation technology. Supported values are platform specific.","title":"isolation"},{"location":"docker/#labels","text":"labels add metadata to containers. You can use either an array or a map. It's recommended that you use reverse-DNS notation to prevent your labels from conflicting with those used by other software. labels: com.example.description: \"Accounting webapp\" com.example.department: \"Finance\" com.example.label-with-empty-value: \"\" labels: - \"com.example.description=Accounting webapp\" - \"com.example.department=Finance\" - \"com.example.label-with-empty-value\" Compose creates containers with canonical labels: com.docker.compose.project set on all resources created by Compose to the user project name com.docker.compose.service set on service containers with service name as defined in the Compose file The com.docker.compose label prefix is reserved. Specifying labels with this prefix in the Compose file results in a runtime error.","title":"labels"},{"location":"docker/#links","text":"Note Availability of the links attribute is implementation specific. links defines a network link to containers in another service. Either specify both the service name and a link alias ( SERVICE:ALIAS ), or just the service name. web: links: - db - db:database - redis Containers for the linked service are reachable at a hostname identical to the alias, or the service name if no alias is specified. Links are not required to enable services to communicate. When no specific network configuration is set, any service is able to reach any other service at that service\u2019s name on the default network. If services do declare networks they are attached to, links does not override the network configuration and services not attached to a shared network are not be able to communicate. Compose doesn't warn you about a configuration mismatch. Links also express implicit dependency between services in the same way as depends_on , so they determine the order of service startup.","title":"links"},{"location":"docker/#logging","text":"logging defines the logging configuration for the service. logging: driver: syslog options: syslog-address: \"tcp://192.168.0.42:123\" The driver name specifies a logging driver for the service's containers. The default and available values are platform specific. Driver specific options can be set with options as key-value pairs.","title":"logging"},{"location":"docker/#network_mode","text":"network_mode sets a service container's network mode. Available values are platform specific, but Compose defines specific values which must be implemented as described if supported: none : Turns off all container networking. host : Gives the container raw access to the host's network interface. service:{name} : Gives the containers access to the specified service only. network_mode: \"host\" network_mode: \"none\" network_mode: \"service:[service name]\" When set, the networks attribute is not allowed and Compose rejects any Compose file containing both attributes.","title":"network_mode"},{"location":"docker/#networks","text":"networks defines the networks that service containers are attached to, referencing entries under the top-level networks key . services: some-service: networks: - some-network - other-network","title":"networks"},{"location":"docker/#aliases","text":"aliases declares alternative hostnames for the service on the network. Other containers on the same network can use either the service name or an alias to connect to one of the service's containers. Since aliases are network-scoped, the same service can have different aliases on different networks. Note A network-wide alias can be shared by multiple containers, and even by multiple services. If it is, then exactly which container the name resolves to is not guaranteed. services: some-service: networks: some-network: aliases: - alias1 - alias3 other-network: aliases: - alias2 In the following example, service frontend is able to reach the backend service at the hostname backend or database on the back-tier network. The service monitoring is able to reach same backend service at backend or mysql on the admin network. services: frontend: image: example/webapp networks: - front-tier - back-tier monitoring: image: example/monitoring networks: - admin backend: image: example/backend networks: back-tier: aliases: - database admin: aliases: - mysql networks: front-tier: back-tier: admin:","title":"aliases"},{"location":"docker/#ipv4_address-ipv6_address","text":"Specify a static IP address for a service container when joining the network. The corresponding network configuration in the top-level networks section must have an ipam attribute with subnet configurations covering each static address. services: frontend: image: example/webapp networks: front-tier: ipv4_address: 172.16.238.10 ipv6_address: 2001:3984:3989::10 networks: front-tier: ipam: driver: default config: - subnet: \"172.16.238.0/24\" - subnet: \"2001:3984:3989::/64\"","title":"ipv4_address, ipv6_address"},{"location":"docker/#link_local_ips","text":"link_local_ips specifies a list of link-local IPs. Link-local IPs are special IPs which belong to a well known subnet and are purely managed by the operator, usually dependent on the architecture where they are deployed. Implementation is platform specific. Example: services: app: image: busybox command: top networks: app_net: link_local_ips: - 57.123.22.11 - 57.123.22.13 networks: app_net: driver: bridge","title":"link_local_ips"},{"location":"docker/#mac_address","text":"mac_address sets the MAC address used by the service container when connecting to this particular network.","title":"mac_address"},{"location":"docker/#priority","text":"priority indicates in which order Compose connects the service\u2019s containers to its networks. If unspecified, the default value is 0. In the following example, the app service connects to app_net_1 first as it has the highest priority. It then connects to app_net_3 , then app_net_2 , which uses the default priority value of 0. services: app: image: busybox command: top networks: app_net_1: priority: 1000 app_net_2: app_net_3: priority: 100 networks: app_net_1: app_net_2: app_net_3:","title":"priority"},{"location":"docker/#mac_address_1","text":"mac_address sets a MAC address for the service container. Note Container runtimes might reject this value (ie. Docker Engine >= v25.0). In that case, you should use networks.mac_address instead.","title":"mac_address"},{"location":"docker/#mem_limit","text":"DEPRECATED: use deploy.limits.memory","title":"mem_limit"},{"location":"docker/#mem_reservation","text":"DEPRECATED: use deploy.reservations.memory","title":"mem_reservation"},{"location":"docker/#mem_swappiness","text":"mem_swappiness defines as a percentage, a value between 0 and 100, for the host kernel to swap out anonymous memory pages used by a container. 0 : Turns off anonymous page swapping. 100 : Sets all anonymous pages as swappable. The default value is platform specific.","title":"mem_swappiness"},{"location":"docker/#memswap_limit","text":"memswap_limit defines the amount of memory the container is allowed to swap to disk. This is a modifier attribute that only has meaning if memory is also set. Using swap lets the container write excess memory requirements to disk when the container has exhausted all the memory that is available to it. There is a performance penalty for applications that swap memory to disk often. If memswap_limit is set to a positive integer, then both memory and memswap_limit must be set. memswap_limit represents the total amount of memory and swap that can be used, and memory controls the amount used by non-swap memory. So if memory =\"300m\" and memswap_limit =\"1g\", the container can use 300m of memory and 700m (1g - 300m) swap. If memswap_limit is set to 0, the setting is ignored, and the value is treated as unset. If memswap_limit is set to the same value as memory , and memory is set to a positive integer, the container does not have access to swap. If memswap_limit is unset, and memory is set, the container can use as much swap as the memory setting, if the host container has swap memory configured. For instance, if memory =\"300m\" and memswap_limit is not set, the container can use 600m in total of memory and swap. If memswap_limit is explicitly set to -1, the container is allowed to use unlimited swap, up to the amount available on the host system.","title":"memswap_limit"},{"location":"docker/#oom_kill_disable","text":"If oom_kill_disable is set, Compose configures the platform so it won't kill the container in case of memory starvation.","title":"oom_kill_disable"},{"location":"docker/#oom_score_adj","text":"oom_score_adj tunes the preference for containers to be killed by platform in case of memory starvation. Value must be within [-1000,1000] range.","title":"oom_score_adj"},{"location":"docker/#pid","text":"pid sets the PID mode for container created by Compose. Supported values are platform specific.","title":"pid"},{"location":"docker/#pids_limit","text":"DEPRECATED: use deploy.resources.limits.pids pids_limit tunes a container\u2019s PIDs limit. Set to -1 for unlimited PIDs. pids_limit: 10","title":"pids_limit"},{"location":"docker/#platform","text":"platform defines the target platform the containers for the service run on. It uses the os[/arch[/variant]] syntax. The values of os , arch , and variant must conform to the convention used by the OCI Image Spec . Compose uses this attribute to determine which version of the image is pulled and/or on which platform the service\u2019s build is performed. platform: darwin platform: windows/amd64 platform: linux/arm64/v8","title":"platform"},{"location":"docker/#ports","text":"Exposes container ports. Note Port mapping must not be used with network_mode: host otherwise a runtime error occurs.","title":"ports"},{"location":"docker/#short-syntax_3","text":"The short syntax is a colon-separated string to set the host IP, host port, and container port in the form: [HOST:]CONTAINER[/PROTOCOL] where: HOST is [IP:](port | range) CONTAINER is port | range PROTOCOL to restrict port to specified protocol. tcp and udp values are defined by the Specification, Compose offers support for platform-specific protocol names. If host IP is not set, it binds to all network interfaces. Ports can be either a single value or a range. Host and container must use equivalent ranges. Either specify both ports ( HOST:CONTAINER ), or just the container port. In the latter case, the container runtime automatically allocates any unassigned port of the host. HOST:CONTAINER should always be specified as a (quoted) string, to avoid conflicts with yaml base-60 float . Examples: ports: - \"3000\" - \"3000-3005\" - \"8000:8000\" - \"9090-9091:8080-8081\" - \"49100:22\" - \"8000-9000:80\" - \"127.0.0.1:8001:8001\" - \"127.0.0.1:5000-5010:5000-5010\" - \"6060:6060/udp\" Note If Host IP mapping is not supported by a container engine, Compose rejects the Compose file and ignores the specified host IP.","title":"Short syntax"},{"location":"docker/#long-syntax_3","text":"The long form syntax allows the configuration of additional fields that can't be expressed in the short form. target : The container port published : The publicly exposed port. It is defined as a string and can be set as a range using syntax start-end . It means the actual port is assigned a remaining available port, within the set range. host_ip : The Host IP mapping, unspecified means all network interfaces ( 0.0.0.0 ). protocol : The port protocol ( tcp or udp ). Defaults to tcp . mode : host : For publishing a host port on each node, or ingress for a port to be load balanced. Defaults to ingress . name : A human-readable name for the port, used to document it's usage within the service ports: - name: http target: 80 host_ip: 127.0.0.1 published: \"8080\" protocol: tcp mode: host - name: https target: 443 host_ip: 127.0.0.1 published: \"8083-9000\" protocol: tcp mode: host","title":"Long syntax"},{"location":"docker/#privileged","text":"privileged configures the service container to run with elevated privileges. Support and actual impacts are platform specific.","title":"privileged"},{"location":"docker/#profiles","text":"profiles defines a list of named profiles for the service to be enabled under. If unassigned, the service is always started but if assigned, it is only started if the profile is activated. If present, profiles follow the regex format of [a-zA-Z0-9][a-zA-Z0-9_.-]+ . services: frontend: image: frontend profiles: [\"frontend\"] phpmyadmin: image: phpmyadmin depends_on: - db profiles: - debug","title":"profiles"},{"location":"docker/#pull_policy","text":"pull_policy defines the decisions Compose makes when it starts to pull images. Possible values are: always : Compose always pulls the image from the registry. never : Compose doesn't pull the image from a registry and relies on the platform cached image. If there is no cached image, a failure is reported. missing : Compose pulls the image only if it's not available in the platform cache. This is the default option if you are not also using the Compose Build Specification . if_not_present is considered an alias for this value for backward compatibility. build : Compose builds the image. Compose rebuilds the image if it's already present.","title":"pull_policy"},{"location":"docker/#read_only","text":"read_only configures the service container to be created with a read-only filesystem.","title":"read_only"},{"location":"docker/#restart","text":"restart defines the policy that the platform applies on container termination. no : The default restart policy. It does not restart the container under any circumstances. always : The policy always restarts the container until its removal. on-failure : The policy restarts the container if the exit code indicates an error. unless-stopped : The policy restarts the container irrespective of the exit code but stops restarting when the service is stopped or removed. restart: \"no\" restart: always restart: on-failure restart: unless-stopped","title":"restart"},{"location":"docker/#runtime","text":"runtime specifies which runtime to use for the service\u2019s containers. The value of runtime is specific to the implementation. For example, runtime can be the name of an implementation of OCI Runtime Spec , such as \"runc\". web: image: busybox:latest command: true runtime: runc","title":"runtime"},{"location":"docker/#scale","text":"scale specifies the default number of containers to deploy for this service. When both are set, scale must be consistent with the replicas attribute in the Deploy Specification .","title":"scale"},{"location":"docker/#secrets","text":"secrets grants access to sensitive data defined by secrets on a per-service basis. Two different syntax variants are supported; the short syntax and the long syntax. Compose reports an error if the secret doesn't exist on the platform or isn't defined in the secrets section of the Compose file. Services can be granted access to multiple secrets. Long and short syntax for secrets may be used in the same Compose file. Defining a secret in the top-level secrets must not imply granting any service access to it. Such grant must be explicit within service specification as secrets service element.","title":"secrets"},{"location":"docker/#short-syntax_4","text":"The short syntax variant only specifies the secret name. This grants the container access to the secret and mounts it as read-only to /run/secrets/ within the container. The source name and destination mountpoint are both set to the secret name. The following example uses the short syntax to grant the frontend service access to the server-certificate secret. The value of server-certificate is set to the contents of the file ./server.cert . services: frontend: image: example/webapp secrets: - server-certificate secrets: server-certificate: file: ./server.cert","title":"Short syntax"},{"location":"docker/#long-syntax_4","text":"The long syntax provides more granularity in how the secret is created within the service's containers. source : The name of the secret as it exists on the platform. target : The name of the file to be mounted in /run/secrets/ in the service's task container, or absolute path of the file if an alternate location is required. Defaults to source if not specified. uid and gid : The numeric UID or GID that owns the file within /run/secrets/ in the service's task containers. Default value is USER running container. mode : The permissions for the file to be mounted in /run/secrets/ in the service's task containers, in octal notation. The default value is world-readable permissions (mode 0444 ). The writable bit must be ignored if set. The executable bit may be set. Note that the uid , gid , and mode attributes are implementation specific. The following example sets the name of the server-certificate secret file to server.crt within the container, sets the mode to 0440 (group-readable), and sets the user and group to 103 . The value of server-certificate secret is provided by the platform through a lookup and the secret's lifecycle is not directly managed by Compose. services: frontend: image: example/webapp secrets: - source: server-certificate target: server.cert uid: \"103\" gid: \"103\" mode: 0440 secrets: server-certificate: external: true","title":"Long syntax"},{"location":"docker/#security_opt","text":"security_opt overrides the default labeling scheme for each container. security_opt: - label:user:USER - label:role:ROLE For further default labeling schemes you can override, see Security configuration .","title":"security_opt"},{"location":"docker/#shm_size","text":"shm_size configures the size of the shared memory ( /dev/shm partition on Linux) allowed by the service container. It's specified as a byte value .","title":"shm_size"},{"location":"docker/#stdin_open","text":"stdin_open configures a service containers to run with an allocated stdin.","title":"stdin_open"},{"location":"docker/#stop_grace_period","text":"stop_grace_period specifies how long Compose must wait when attempting to stop a container if it doesn't handle SIGTERM (or whichever stop signal has been specified with stop_signal ), before sending SIGKILL. It's specified as a duration . stop_grace_period: 1s stop_grace_period: 1m30s Default value is 10 seconds for the container to exit before sending SIGKILL.","title":"stop_grace_period"},{"location":"docker/#stop_signal","text":"stop_signal defines the signal that Compose uses to stop the service containers. If unset containers are stopped by Compose by sending SIGTERM . stop_signal: SIGUSR1","title":"stop_signal"},{"location":"docker/#storage_opt","text":"storage_opt defines storage driver options for a service. storage_opt: size: '1G'","title":"storage_opt"},{"location":"docker/#sysctls","text":"sysctls defines kernel parameters to set in the container. sysctls can use either an array or a map. sysctls: net.core.somaxconn: 1024 net.ipv4.tcp_syncookies: 0 sysctls: - net.core.somaxconn=1024 - net.ipv4.tcp_syncookies=0 You can only use sysctls that are namespaced in the kernel. Docker does not support changing sysctls inside a container that also modify the host system. For an overview of supported sysctls, refer to configure namespaced kernel parameters (sysctls) at runtime .","title":"sysctls"},{"location":"docker/#tmpfs","text":"tmpfs mounts a temporary file system inside the container. It can be a single value or a list. tmpfs: /run tmpfs: - /run - /tmp","title":"tmpfs"},{"location":"docker/#tty","text":"tty configures service container to run with a TTY.","title":"tty"},{"location":"docker/#ulimits","text":"ulimits overrides the default ulimits for a container. It's specified either as an integer for a single limit or as mapping for soft/hard limits. ulimits: nproc: 65535 nofile: soft: 20000 hard: 40000","title":"ulimits"},{"location":"docker/#user","text":"user overrides the user used to run the container process. The default is set by the image (i.e. Dockerfile USER ). If it's not set, then root .","title":"user"},{"location":"docker/#userns_mode","text":"userns_mode sets the user namespace for the service. Supported values are platform specific and may depend on platform configuration. userns_mode: \"host\"","title":"userns_mode"},{"location":"docker/#volumes","text":"volumes define mount host paths or named volumes that are accessible by service containers. You can use volumes to define multiple types of mounts; volume , bind , tmpfs , or npipe . If the mount is a host path and is only used by a single service, it can be declared as part of the service definition. To reuse a volume across multiple services, a named volume must be declared in the top-level volumes key . The following example shows a named volume ( db-data ) being used by the backend service, and a bind mount defined for a single service. services: backend: image: example/backend volumes: - type: volume source: db-data target: /data volume: nocopy: true - type: bind source: /var/run/postgres/postgres.sock target: /var/run/postgres/postgres.sock volumes: db-data:","title":"volumes"},{"location":"docker/#short-syntax_5","text":"The short syntax uses a single string with colon-separated values to specify a volume mount ( VOLUME:CONTAINER_PATH ), or an access mode ( VOLUME:CONTAINER_PATH:ACCESS_MODE ). VOLUME : Can be either a host path on the platform hosting containers (bind mount) or a volume name. CONTAINER_PATH : The path in the container where the volume is mounted. ACCESS_MODE : A comma-separated , list of options: rw : Read and write access. This is the default if none is specified. ro : Read-only access. z : SELinux option indicating that the bind mount host content is shared among multiple containers. Z : SELinux option indicating that the bind mount host content is private and unshared for other containers. Note The SELinux re-labeling bind mount option is ignored on platforms without SELinux. Note Relative host paths are only supported by Compose that deploy to a local container runtime. This is because the relative path is resolved from the Compose file\u2019s parent directory which is only applicable in the local case. When Compose deploys to a non-local platform it rejects Compose files which use relative host paths with an error. To avoid ambiguities with named volumes, relative paths should always begin with . or .. .","title":"Short syntax"},{"location":"docker/#long-syntax_5","text":"The long form syntax allows the configuration of additional fields that can't be expressed in the short form. type : The mount type. Either volume , bind , tmpfs , npipe , or cluster source : The source of the mount, a path on the host for a bind mount, or the name of a volume defined in the top-level volumes key . Not applicable for a tmpfs mount. target : The path in the container where the volume is mounted. read_only : Flag to set the volume as read-only. bind : Used to configure additional bind options: propagation : The propagation mode used for the bind. create_host_path : Creates a directory at the source path on host if there is nothing present. Compose does nothing if there is something present at the path. This is automatically implied by short syntax for backward compatibility with docker-compose legacy. selinux : The SELinux re-labeling option z (shared) or Z (private) volume : Configures additional volume options: nocopy : Flag to disable copying of data from a container when a volume is created. tmpfs : Configures additional tmpfs options: size : The size for the tmpfs mount in bytes (either numeric or as bytes unit). mode : The file mode for the tmpfs mount as Unix permission bits as an octal number. consistency : The consistency requirements of the mount. Available values are platform specific.","title":"Long syntax"},{"location":"docker/#volumes_from","text":"volumes_from mounts all of the volumes from another service or container. You can optionally specify read-only access ro or read-write rw . If no access level is specified, then read-write access is used. You can also mount volumes from a container that is not managed by Compose by using the container: prefix. volumes_from: - service_name - service_name:ro - container:container_name - container:container_name:rw","title":"volumes_from"},{"location":"docker/#working_dir","text":"working_dir overrides the container's working directory which is specified by the image, for example Dockerfile's WORKDIR .","title":"working_dir"},{"location":"docker/#networks-top-level-element","text":"Networks are the layer that allow services to communicate with each other. The top-level networks element lets you configure named networks that can be reused across multiple services. To use a network across multiple services, you must explicitly grant each service access by using the networks attribute within the services top-level element. The networks top-level element has additional syntax that provides more granular control.","title":"Networks top-level element"},{"location":"docker/#examples","text":"","title":"Examples"},{"location":"docker/#basic-example","text":"In the following example, at runtime, networks front-tier and back-tier are created and the frontend service is connected to front-tier and back-tier networks. services: frontend: image: example/webapp networks: - front-tier - back-tier networks: front-tier: back-tier:","title":"Basic example"},{"location":"docker/#advanced-example","text":"services: proxy: build: ./proxy networks: - frontend app: build: ./app networks: - frontend - backend db: image: postgres networks: - backend networks: frontend: ## Use a custom driver driver: custom-driver-1 backend: ## Use a custom driver which takes special options driver: custom-driver-2 driver_opts: foo: \"1\" bar: \"2\" The advanced example shows a Compose file which defines two custom networks. The proxy service is isolated from the db service, because they do not share a network in common. Only app can talk to both.","title":"Advanced example"},{"location":"docker/#attributes","text":"","title":"Attributes"},{"location":"docker/#driver","text":"driver specifies which driver should be used for this network. Compose returns an error if the driver is not available on the platform. networks: db-data: driver: overlay Default and available values are platform specific. Compose supports the following drivers: none and host host : Use the host's networking stack. none : Turn off networking.","title":"driver"},{"location":"docker/#host-or-none","text":"The syntax for using built-in networks such as host and none is different, as such networks implicitly exist outside the scope of Compose. To use them, you must define an external network with the name host or none and an alias that Compose can use ( hostnet and nonet in the following example), then grant the service access to that network using its alias. services: web: networks: hostnet: {} networks: hostnet: external: true name: host services: web: ... networks: nonet: {} networks: nonet: external: true name: none","title":"host or none"},{"location":"docker/#driver_opts","text":"driver_opts specifies a list of options as key-value pairs to pass to the driver. These options are driver-dependent. Consult the driver's documentation for more information. networks: db-data: driver_opts: foo: \"bar\" baz: 1","title":"driver_opts"},{"location":"docker/#attachable","text":"If attachable is set to true , then standalone containers should be able to attach to this network, in addition to services. If a standalone container attaches to the network, it can communicate with services and other standalone containers that are also attached to the network. networks: mynet1: driver: overlay attachable: true","title":"attachable"},{"location":"docker/#enable_ipv6","text":"enable_ipv6 enables IPv6 networking. For an example, see step four of Create an IPv6 network .","title":"enable_ipv6"},{"location":"docker/#external","text":"If set to true : - external specifies that this network\u2019s lifecycle is maintained outside of that of the application. Compose doesn't attempt to create these networks, and returns an error if one doesn't exist. - All other attributes apart from name are irrelevant. If Compose detects any other attribute, it rejects the Compose file as invalid. In the example below, proxy is the gateway to the outside world. Instead of attempting to create a network, Compose queries the platform for an existing network simply called outside and connects the proxy service's containers to it. services: proxy: image: example/proxy networks: - outside - default app: image: example/app networks: - default networks: outside: external: true","title":"external"},{"location":"docker/#ipam","text":"ipam specifies a custom IPAM configuration. This is an object with several properties, each of which is optional: driver : Custom IPAM driver, instead of the default. config : A list with zero or more configuration elements, each containing a: subnet : Subnet in CIDR format that represents a network segment ip_range : Range of IPs from which to allocate container IPs gateway : IPv4 or IPv6 gateway for the master subnet aux_addresses : Auxiliary IPv4 or IPv6 addresses used by Network driver, as a mapping from hostname to IP options : Driver-specific options as a key-value mapping. networks: mynet1: ipam: driver: default config: - subnet: 172.28.0.0/16 ip_range: 172.28.5.0/24 gateway: 172.28.5.254 aux_addresses: host1: 172.28.1.5 host2: 172.28.1.6 host3: 172.28.1.7 options: foo: bar baz: \"0\"","title":"ipam"},{"location":"docker/#internal","text":"By default, Compose provides external connectivity to networks. internal , when set to true , allows you to create an externally isolated network.","title":"internal"},{"location":"docker/#labels_1","text":"Add metadata to containers using labels . You can use either an array or a dictionary. It is recommended that you use reverse-DNS notation to prevent labels from conflicting with those used by other software. networks: mynet1: labels: com.example.description: \"Financial transaction network\" com.example.department: \"Finance\" com.example.label-with-empty-value: \"\" networks: mynet1: labels: - \"com.example.description=Financial transaction network\" - \"com.example.department=Finance\" - \"com.example.label-with-empty-value\" Compose sets com.docker.compose.project and com.docker.compose.network labels.","title":"labels"},{"location":"docker/#name","text":"name sets a custom name for the network. The name field can be used to reference networks which contain special characters. The name is used as is and is not scoped with the project name. networks: network1: name: my-app-net It can also be used in conjunction with the external property to define the platform network that Compose should retrieve, typically by using a parameter so the Compose file doesn't need to hard-code runtime specific values: networks: network1: external: true name: \"${NETWORK_ID}\"","title":"name"},{"location":"docker/#volumes-top-level-element","text":"Volumes are persistent data stores implemented by the container engine. Compose offers a neutral way for services to mount volumes, and configuration parameters to allocate them to infrastructure. The top-level volumes declaration lets you configure named volumes that can be reused across multiple services. To use a volume across multiple services, you must explicitly grant each service access by using the volumes attribute within the services top-level element. The volumes attribute has additional syntax that provides more granular control.","title":"Volumes top-level element"},{"location":"docker/#example","text":"The following example shows a two-service setup where a database's data directory is shared with another service as a volume, named db-data , so that it can be periodically backed up. services: backend: image: example/database volumes: - db-data:/etc/data backup: image: backup-service volumes: - db-data:/var/lib/backup/data volumes: db-data: The db-data volume is mounted at the /var/lib/backup/data and /etc/data container paths for backup and backend respectively. Running docker compose up creates the volume if it doesn't already exist. Otherwise, the existing volume is used and is recreated if it's manually deleted outside of Compose.","title":"Example"},{"location":"docker/#attributes_1","text":"An entry under the top-level volumes section can be empty, in which case it uses the container engine's default configuration for creating a volume. Optionally, you can configure it with the following keys:","title":"Attributes"},{"location":"docker/#driver_1","text":"Specifies which volume driver should be used. Default and available values are platform specific. If the driver is not available, Compose returns an error and doesn't deploy the application. volumes: db-data: driver: foobar","title":"driver"},{"location":"docker/#driver_opts_1","text":"driver_opts specifies a list of options as key-value pairs to pass to the driver for this volume. The options are driver-dependent. volumes: example: driver_opts: type: \"nfs\" o: \"addr=10.40.0.199,nolock,soft,rw\" device: \":/docker/example\"","title":"driver_opts"},{"location":"docker/#external_1","text":"If set to true : - external specifies that this volume already exists on the platform and its lifecycle is managed outside of that of the application. Compose doesn't then create the volume, and returns an error if the volume doesn't exist. - All other attributes apart from name are irrelevant. If Compose detects any other attribute, it rejects the Compose file as invalid. In the example below, instead of attempting to create a volume called {project_name}_db-data , Compose looks for an existing volume simply called db-data and mounts it into the backend service's containers. services: backend: image: example/database volumes: - db-data:/etc/data volumes: db-data: external: true","title":"external"},{"location":"docker/#labels_2","text":"labels are used to add metadata to volumes. You can use either an array or a dictionary. It's recommended that you use reverse-DNS notation to prevent your labels from conflicting with those used by other software. volumes: db-data: labels: com.example.description: \"Database volume\" com.example.department: \"IT/Ops\" com.example.label-with-empty-value: \"\" volumes: db-data: labels: - \"com.example.description=Database volume\" - \"com.example.department=IT/Ops\" - \"com.example.label-with-empty-value\" Compose sets com.docker.compose.project and com.docker.compose.volume labels.","title":"labels"},{"location":"docker/#name_1","text":"name sets a custom name for a volume. The name field can be used to reference volumes that contain special characters. The name is used as is and is not scoped with the stack name. volumes: db-data: name: \"my-app-data\" This makes it possible to make this lookup name a parameter of the Compose file, so that the model ID for the volume is hard-coded but the actual volume ID on the platform is set at runtime during deployment. For example, if DATABASE_VOLUME=my_volume_001 in your .env file: volumes: db-data: name: ${DATABASE_VOLUME} Running docker compose up uses the volume called my_volume_001 . It can also be used in conjunction with the external property. This means the name of the volume used to lookup the actual volume on the platform is set separately from the name used to refer to it within the Compose file: volumes: db-data: external: name: actual-name-of-volume","title":"name"},{"location":"docker/#configs-top-level-element","text":"Configs allow services to adapt their behaviour without the need to rebuild a Docker image. Services can only access configs when explicitly granted by a configs attribute within the services top-level element. As with volumes, configs are mounted as files into a service's container's filesystem. The location of the mount point within the container defaults to / in Linux containers and C:\\ in Windows containers. By default, the config: - Is owned by the user running the container command but can be overridden by service configuration. - Has world-readable permissions (mode 0444), unless the service is configured to override this. The top-level configs declaration defines or references configuration data that is granted to services in your Compose application. The source of the config is either file or external . file : The config is created with the contents of the file at the specified path. environment : The config content is created with the value of an environment variable. content : The content is created with the inlined value. external : If set to true, external specifies that this config has already been created. Compose does not attempt to create it, and if it does not exist, an error occurs. name : The name of the config object in the container engine to look up. This field can be used to reference configs that contain special characters. The name is used as is and will not be scoped with the project name.","title":"Configs top-level element"},{"location":"docker/#example-1","text":"_http_config is created when the application is deployed, by registering the content of the httpd.conf as the configuration data. configs: http_config: file: ./httpd.conf Alternatively, http_config can be declared as external. Compose looks up http_config to expose the configuration data to relevant services. configs: http_config: external: true","title":"Example 1"},{"location":"docker/#example-2","text":"_app_config is created when the application is deployed, by registering the inlined content as the configuration data. This comes with the benefits Compose will infer variables when creating the config, which allows to adjust content according to service configuration: configs: app_config: content: | debug=${DEBUG} spring.application.admin.enabled=${DEBUG} spring.application.name=${COMPOSE_PROJECT_NAME}","title":"Example 2"},{"location":"docker/#example-3","text":"External configs lookup can also use a distinct key by specifying a name . The following example modifies the previous one to look up a config using the parameter HTTP_CONFIG_KEY . The the actual lookup key will is set at deployment time by the interpolation of variables, but exposed to containers as hard-coded ID http_config . configs: http_config: external: true name: \"${HTTP_CONFIG_KEY}\" If external is set to true , all other attributes apart from name are irrelevant. If Compose detecs any other attribute, it rejects the Compose file as invalid.","title":"Example 3"},{"location":"docker/#secrets-top-level-element","text":"Secrets are a flavor of Configs focusing on sensitive data, with specific constraint for this usage. Services can only access secrets when explicitly granted by a secrets attribute within the services top-level element. The top-level secrets declaration defines or references sensitive data that is granted to the services in your Compose application. The source of the secret is either file or environment . file : The secret is created with the contents of the file at the specified path. environment : The secret is created with the value of an environment variable. external : If set to true, external specifies that this secret has already been created. Compose does not attempt to create it, and if it does not exist, an error occurs. name : The name of the secret object in Docker. This field can be used to reference secrets that contain special characters. The name is used as is and isn't scoped with the project name.","title":"Secrets top-level element"},{"location":"docker/#example-1_1","text":"server-certificate secret is created as _server-certificate when the application is deployed, by registering content of the server.cert as a platform secret. secrets: server-certificate: file: ./server.cert","title":"Example 1"},{"location":"docker/#example-2_1","text":"token secret is created as _token when the application is deployed, by registering the content of the OAUTH_TOKEN environment variable as a platform secret. secrets: token: environment: \"OAUTH_TOKEN\" Alternatively, server-certificate can be declared as external. Compose looks up the server-certificate secret to expose to relevant services. secrets: server-certificate: external: true","title":"Example 2"},{"location":"docker/#example-3_1","text":"External secrets lookup can also use a distinct key by specifying a name . The following example modifies the previous example to look up a secret using the name CERTIFICATE_KEY . The actual lookup key is set at deployment time by the interpolation of variables, but exposed to containers as hard-coded ID server-certificate . secrets: server-certificate: external: true name: \"${CERTIFICATE_KEY}\" If external is set to true , all other attributes apart from name are irrelevant. If Compose detects any other attribute, it rejects the Compose file as invalid. Your Compose file needs to explicitly grant access to the secrets to relevant services in your application.","title":"Example 3"},{"location":"docker/#fragments","text":"With Compose, you can use built-in YAML features to make your Compose file neater and more efficient. Anchors and aliases let you create re-usable blocks. This is useful if you start to find common configurations that span multiple services. Having re-usable blocks minimizes potential mistakes. Anchors are created using the & sign. The sign is followed by an alias name. You can use this alias with the * sign later to reference the value following the anchor. Make sure there is no space between the & and the * characters and the following alias name. You can use more than one anchor and alias in a single Compose file.","title":"Fragments"},{"location":"docker/#example-1_2","text":"volumes: db-data: &default-volume driver: default metrics: *default-volume In the example above, a default-volume anchor is created based on the db-data volume. It is later reused by the alias *default-volume to define the metrics volume. Anchor resolution takes place before variables interpolation , so variables can't be used to set anchors or aliases.","title":"Example 1"},{"location":"docker/#example-2_2","text":"services: first: image: my-image:latest environment: &env - CONFIG_KEY - EXAMPLE_KEY - DEMO_VAR second: image: another-image:latest environment: *env If you have an anchor that you want to use in more than one service, use it in conjunction with an extension to make your Compose file easier to maintain.","title":"Example 2"},{"location":"docker/#example-3_2","text":"You may want to partially override values. Compose follows the rule outlined by YAML merge type . In the following example, metrics volume specification uses alias to avoid repetition but overrides name attribute: services: backend: image: example/database volumes: - db-data - metrics volumes: db-data: &default-volume driver: default name: \"data\" metrics: <<: *default-volume name: \"metrics\"","title":"Example 3"},{"location":"docker/#example-4","text":"You can also extend the anchor to add additional values. services: first: image: my-image:latest environment: &env FOO: BAR ZOT: QUIX second: image: another-image:latest environment: <<: *env YET_ANOTHER: VARIABLE Note YAML merge only applies to mappings, and can't be used with sequences. In example above, the environment variables must be declared using the FOO: BAR mapping syntax, while the sequence syntax - FOO=BAR is only valid when no fragments are involved.","title":"Example 4"},{"location":"docker/#extension","text":"As with Fragments , Extensions can be used to make your Compose file more efficient and easier to maintain. Extensions can also be used with anchors and aliases . Use the prefix x- as a top-level element to modularize configurations that you want to reuse. Compose ignores any fields that start with x- , this is the sole exception where Compose silently ignores unrecognized fields. They also can be used within any structure in a Compose file where user-defined keys are not expected. Compose use those to enable experimental features, the same way browsers add support for custom CSS features","title":"Extension"},{"location":"docker/#example-1_3","text":"x-custom: foo: - bar - zot services: webapp: image: example/webapp x-foo: bar service: backend: deploy: placement: x-aws-role: \"arn:aws:iam::XXXXXXXXXXXX:role/foo\" x-aws-region: \"eu-west-3\" x-azure-region: \"france-central\"","title":"Example 1"},{"location":"docker/#example-2_3","text":"x-env: &env environment: - CONFIG_KEY - EXAMPLE_KEY services: first: <<: *env image: my-image:latest second: <<: *env image: another-image:latest In this example, the environment variables do not belong to either of the services. They\u2019ve been lifted out completely into the x-env extension field. This defines a new node which contains the environment field. The &env YAML anchor is used so both services can reference the extension field\u2019s value as *env .","title":"Example 2"},{"location":"docker/#example-3_3","text":"x-function: &function labels: function: \"true\" depends_on: - gateway networks: - functions deploy: placement: constraints: - 'node.platform.os == linux' services: ## Node.js gives OS info about the node (Host) nodeinfo: <<: *function image: functions/nodeinfo:latest environment: no_proxy: \"gateway\" https_proxy: $https_proxy ## Uses `cat` to echo back response, fastest function to execute. echoit: <<: *function image: functions/alpine:health environment: fprocess: \"cat\" no_proxy: \"gateway\" https_proxy: $https_proxy The nodeinfo and echoit services both include the x-function extension via the &function anchor, then set their specific image and environment.","title":"Example 3"},{"location":"docker/#example-4_1","text":"Using YAML merge it is also possible to use multiple extensions and share and override additional attributes for specific needs: x-environment: &default-environment FOO: BAR ZOT: QUIX x-keys: &keys KEY: VALUE services: frontend: image: example/webapp environment: << : [*default-environment, *keys] YET_ANOTHER: VARIABLE Note YAML merge only applies to mappings, and can't be used with sequences. In the example above, the environment variables are declared using the FOO: BAR mapping syntax, while the sequence syntax - FOO=BAR is only valid when no fragments are involved.","title":"Example 4"},{"location":"docker/#informative-historical-notes","text":"This section is informative. At the time of writing, the following prefixes are known to exist: Prefix Vendor/Organization docker Docker kubernetes Kubernetes","title":"Informative Historical Notes"},{"location":"docker/#specifying-byte-values","text":"Values express a byte value as a string in {amount}{byte unit} format: The supported units are b (bytes), k or kb (kilo bytes), m or mb (mega bytes) and g or gb (giga bytes). 2b 1024kb 2048k 300m 1gb","title":"Specifying byte values"},{"location":"docker/#specifying-durations","text":"Values express a duration as a string in the form of {value}{unit} . The supported units are us (microseconds), ms (milliseconds), s (seconds), m (minutes) and h (hours). Values can combine multiple values without separator. 10ms 40s 1m30s 1h5m30s20ms","title":"Specifying durations"},{"location":"docker/#interpolation","text":"Values in a Compose file can be set by variables and interpolated at runtime. Compose files use a Bash-like syntax ${VARIABLE} . Both $VARIABLE and ${VARIABLE} syntax is supported. Default values can be defined inline using typical shell syntax: ${VARIABLE:-default} evaluates to default if VARIABLE is unset or empty in the environment. ${VARIABLE-default} evaluates to default only if VARIABLE is unset in the environment. Similarly, the following syntax allows you to specify mandatory variables: ${VARIABLE:?err} exits with an error message containing err if VARIABLE is unset or empty in the environment. ${VARIABLE?err} exits with an error message containing err only if VARIABLE is unset in the environment. Interpolation can also be nested: ${VARIABLE:-${FOO}} ${VARIABLE?$FOO} ${VARIABLE:-${FOO:-default}} Other extended shell-style features, such as ${VARIABLE/foo/bar} , are not supported by Compose. You can use a $$ (double-dollar sign) when your configuration needs a literal dollar sign. This also prevents Compose from interpolating a value, so a $$ allows you to refer to environment variables that you don't want processed by Compose. web: build: . command: \"$$VAR_NOT_INTERPOLATED_BY_COMPOSE\" If Compose can't resolve a substituted variable and no default value is defined, it displays a warning and substitutes the variable with an empty string. As any values in a Compose file can be interpolated with variable substitution, including compact string notation for complex elements, interpolation is applied before a merge on a per-file basis. Interpolation applies only to YAML values, not to keys. For the few places where keys are actually arbitrary user-defined strings, such as labels or environment , an alternate equal sign syntax must be used for interpolation to apply. For example: services: foo: labels: \"$VAR_NOT_INTERPOLATED_BY_COMPOSE\": \"BAR\" services: foo: labels: - \"$VAR_INTERPOLATED_BY_COMPOSE=BAR\"","title":"Interpolation"},{"location":"docker/#merge-and-override","text":"Compose lets you define a Compose application model through multiple Compose files . When doing so, Compose follows the rules declared in this section to merge Compose files.","title":"Merge and override"},{"location":"docker/#mapping","text":"A YAML mapping gets merged by adding missing entries and merging the conflicting ones. Merging the following example YAML trees: services: foo: key1: value1 key2: value2 services: foo: key2: VALUE key3: value3 Results in a Compose application model equivalent to the YAML tree: services: foo: key1: value1 key2: VALUE key3: value3","title":"Mapping"},{"location":"docker/#sequence","text":"A YAML sequence is merged by appending values from the overriding Compose file to the previous one. Merging the following example YAML trees: services: foo: DNS: - 1.1.1.1 services: foo: DNS: - 8.8.8.8 Results in a Compose application model equivalent to the YAML tree: services: foo: DNS: - 1.1.1.1 - 8.8.8.8","title":"Sequence"},{"location":"docker/#exceptions","text":"","title":"Exceptions"},{"location":"docker/#shell-commands","text":"When merging Compose files that use the services attributes command , entrypoint and healthcheck: test , the value is overridden by the latest Compose file, and not appended. Merging the following example YAML trees: services: foo: command: [\"echo\", \"foo\"] services: foo: command: [\"echo\", \"bar\"] Results in a Compose application model equivalent to the YAML tree: services: foo: command: [\"echo\", \"bar\"]","title":"Shell commands"},{"location":"docker/#unique-resources","text":"Applies to the ports , volumes , secrets and configs services attributes. While these types are modeled in a Compose file as a sequence, they have special uniqueness requirements: Attribute Unique key volumes target secrets source configs source ports {ip, target, published, protocol} When merging Compose files, Compose appends new entries that do not violate a uniqueness constraint and merge entries that share a unique key. Merging the following example YAML trees: services: foo: volumes: - foo:/work services: foo: volumes: - bar:/work Results in a Compose application model equivalent to the YAML tree: services: foo: volumes: - bar:/work","title":"Unique resources"},{"location":"docker/#reset-value","text":"In addition to the previously described mechanism, an override Compose file can also be used to remove elements from your application model. For this purpose, the custom YAML tag !reset can be set to override a value set by the overriden Compose file. A valid value for attribute must be provided, but will be ignored and target attribute will be set with type's default value or null . For readability, it is recommended to explicitly set the attribute value to the null ( null ) or empty array [] (with !reset null or !reset [] ) so that it is clear that resulting attribute will be cleared. A base compose.yaml file: services: app: image: myapp ports: - \"8080:80\" environment: FOO: BAR And an overide.compose.yaml file: services: app: image: myapp ports: !reset [] environment: FOO: !reset null Results in: services: app: image: myapp","title":"Reset value"},{"location":"docker/#replace-value","text":"While !reset can be used to remove a declaration from a Compose file using an override file, !override allows you to fully replace an attribute, bypassing the standard merge rules. A typical example is to fully replace a resource definition, to rely on a distinct model but using the same name. A base compose.yaml file: services: app: image: myapp ports: - \"8080:80\" To remove the original port, but expose a new one, the following override file is used: services: app: ports: !override - \"8443:443\" This results in: services: app: image: myapp ports: - \"8443:443\" If !override had not been used, both 8080:80 and 8443:443 would be exposed as per the merging rules outlined above .","title":"Replace value"},{"location":"docker/#include","text":"A Compose application can declare dependency on another Compose application. This is useful if: - You want to reuse other Compose files. - You need to factor out parts of your application model into separate Compose files so they can be managed separately or shared with others. - Teams need to keep a Compose file reasonably complicated for the limited amount of resources it has to declare for it's own sub-domain, within a larger deployment. The include top-level section is used to define the dependency on another Compose application, or sub-domain. Each path listed in the include section is loaded as an individual Compose application model, with it's own project directory, in order to resolve relative paths. Once the included Compose application is loaded, all resources definitions are copied into the current Compose application model. Compose displays a warning if resource names conflict and doesn't try to merge them. To enforce this, include is evaluated after the Compose file(s) selected to define the Compose application model have been parsed and merged, so that conflicts between Compose files are detected. include applies recursively so an included Compose file which declares its own include section, triggers those other files to be included as well. Any volumes, networks, or other resources pulled in from the included Compose file can be used by the current Compose application for cross-service references. For example: include: - my-compose-include.yaml #with serviceB declared services: serviceA: build: . depends_on: - serviceB #use serviceB directly as if it was declared in this Compose file Compose also supports the use of interpolated variables with include . It's recommended that you specify mandatory variables . For example: include: -${INCLUDE_PATH:?FOO}/compose.yaml","title":"Include"},{"location":"docker/#short-syntax_6","text":"The short syntax only defines paths to other Compose files. The file is loaded with the parent folder as the project directory, and an optional .env file that is loaded to define any variables' default values by interpolation. The local project's environment can override those values. include: - ../commons/compose.yaml - ../another_domain/compose.yaml services: webapp: depends_on: - included-service ## defined by another_domain In the above example, both ../commons/compose.yaml and ../another_domain/compose.yaml are loaded as individual Compose projects. Relative paths in Compose files being referred by include are resolved relative to their own Compose file path, not based on the local project's directory. Variables are interpolated using values set in the optional .env file in same folder, and is overridden by the local project's environment.","title":"Short syntax"},{"location":"docker/#long-syntax_6","text":"The long syntax offers more control over the sub-project parsing: include: - path: ../commons/compose.yaml project_directory: .. env_file: ../another/.env","title":"Long syntax"},{"location":"docker/#path","text":"path is required and defines the location of the Compose file(s) to be parsed and included into the local Compose model. path can be set either to a string when a single Compose file is involved, or to a list of strings when multiple Compose files need to be merged together to define the Compose model to be included in the local application. include: - path: - ../commons/compose.yaml - ./commons-override.yaml","title":"path"},{"location":"docker/#project_directory","text":"project_directory defines a base path to resolve relative paths set in the Compose file. It defaults to the directory of the included Compose file.","title":"project_directory"},{"location":"docker/#env_file_1","text":"env_file defines an environment file(s) to use to define default values when interpolating variables in the Compose file being parsed. It defaults to .env file in the project_directory for the Compose file being parsed. env_file can be set either to a string or a list of strings when multiple environment files need to be merged to define a project environment. include: - path: ../another/compose.yaml env_file: - ../another/.env - ../another/dev.env The local project's environment has precedence over the values set by the Compose file, so that the local project can override values for customization.","title":"env_file"},{"location":"docker/#profiles_1","text":"With profiles you can define a set of active profiles so your Compose application model is adjusted for various usages and environments. The exact mechanism is implementation specific and may include command line flags, environment variables, etc. The services top-level element supports a profiles attribute to define a list of named profiles. Services without a profiles attribute are always enabled. A service is ignored by Compose when none of the listed profiles match the active ones, unless the service is explicitly targeted by a command. In that case its profile is added to the set of active profiles. Note All other top-level elements are not affected by profiles and are always active. References to other services (by links , extends or shared resource syntax service:xxx ) do not automatically enable a component that would otherwise have been ignored by active profiles. Instead Compose returns an error.","title":"Profiles"},{"location":"docker/#illustrative-example_1","text":"services: foo: image: foo bar: image: bar profiles: - test baz: image: baz depends_on: - bar profiles: - test zot: image: zot depends_on: - bar profiles: - debug In the above example: If the Compose application model is parsed with no profile enabled, it only contains the foo service. If the profile test is enabled, the model contains the services bar and baz , and service foo , which is always enabled. If the profile debug is enabled, the model contains both foo and zot services, but not bar and baz , and as such the model is invalid regarding the depends_on constraint of zot . If the profiles debug and test are enabled, the model contains all services; foo , bar , baz and zot . If Compose is executed with bar as the explicit service to run, bar and the test profile are active even if test profile is not enabled. If Compose is executed with baz as the explicit service to run, the service baz and the profile test are active and bar is pulled in by the depends_on constraint. If Compose is executed with zot as the explicit service to run, again the model is invalid regarding the depends_on constraint of zot , since zot and bar have no common profiles listed. If Compose is executed with zot as the explicit service to run and profile test is enabled, profile debug is automatically enabled and service bar is pulled in as a dependency starting both services zot and bar . See how you can use profiles in Docker Compose .","title":"Illustrative example"},{"location":"embedding/","text":"Embedding Embedding is a way to represent categorical variables in a way that can be used by machine learning algorithms. It is a way to represent a categorical variable as a continuous vector of numbers. This is done by creating a matrix of weights that is learned during the training process. The matrix is initialized randomly and updated during training. The embedding matrix is a dense representation of the categorical variable. Leaderboard https://huggingface.co/spaces/mteb/leaderboard","title":"embedding"},{"location":"embedding/#embedding","text":"Embedding is a way to represent categorical variables in a way that can be used by machine learning algorithms. It is a way to represent a categorical variable as a continuous vector of numbers. This is done by creating a matrix of weights that is learned during the training process. The matrix is initialized randomly and updated during training. The embedding matrix is a dense representation of the categorical variable.","title":"Embedding"},{"location":"embedding/#leaderboard","text":"https://huggingface.co/spaces/mteb/leaderboard","title":"Leaderboard"},{"location":"framework/","text":"Framework Pytorch vs Tensorflow The image range is different for each framework. In PyTorch, the image range is 0-1 while TensorFlow uses a range from 0 to 255. To use TensorFlow, we have to adapt the image range. To TF def dataset_to_tf( dataset, cols_to_retain, collate_fn, collate_fn_args, columns_to_np_types, output_signature, shuffle, batch_size, drop_remainder, ): \"\"\"Create a tf.data.Dataset from the underlying Dataset. This is a single-process method - the multiprocess equivalent is multiprocess_dataset_to_tf. Args: dataset (`Dataset`): Dataset to wrap with tf.data.Dataset. cols_to_retain (`List[str]`): Dataset column(s) to load in the tf.data.Dataset. It is acceptable to include column names that are created by the `collate_fn` and that do not exist in the original dataset. collate_fn(`Callable`): A function or callable object (such as a `DataCollator`) that will collate lists of samples into a batch. collate_fn_args (`Dict`): A `dict` of keyword arguments to be passed to the `collate_fn`. Can be empty. columns_to_np_types (`Dict[str, np.dtype]`): A `dict` mapping column names to numpy dtypes. output_signature (`Dict[str, tf.TensorSpec]`): A `dict` mapping column names to `tf.TensorSpec` objects. shuffle(`bool`): Shuffle the dataset order when loading. Recommended True for training, False for validation/evaluation. batch_size (`int`): Size of batches to load from the dataset. drop_remainder(`bool`, default `None`): Drop the last incomplete batch when loading. If not provided, defaults to the same setting as shuffle. Returns: `tf.data.Dataset` \"\"\" if config.TF_AVAILABLE: import tensorflow as tf else: raise ImportError(\"Called a Tensorflow-specific function but Tensorflow is not installed.\") getter_fn = partial( np_get_batch, dataset=dataset, cols_to_retain=cols_to_retain, collate_fn=collate_fn, collate_fn_args=collate_fn_args, columns_to_np_types=columns_to_np_types, return_dict=False, # TF expects numpy_function to return a list and will not accept a dict ) @tf.function(input_signature=[tf.TensorSpec(None, tf.int64)]) def fetch_function(indices): output = tf.numpy_function( getter_fn, inp=[indices], # This works because dictionaries always output in the same order Tout=[tf.dtypes.as_dtype(dtype) for dtype in columns_to_np_types.values()], ) return {key: output[i] for i, key in enumerate(columns_to_np_types.keys())} tf_dataset = tf.data.Dataset.from_tensor_slices(np.arange(len(dataset), dtype=np.int64)) if shuffle: tf_dataset = tf_dataset.shuffle(len(dataset)) tf_dataset = tf_dataset.batch(batch_size, drop_remainder=drop_remainder).map(fetch_function) def ensure_shapes(input_dict): return {key: tf.ensure_shape(val, output_signature[key].shape) for key, val in input_dict.items()} return tf_dataset.map(ensure_shapes) pytorch dataset to tf dataset import tensorflow as tf import torch # Assume that we have a PyTorch Dataset object called 'dataset' def pytorch_dataset_to_tensorflow_dataset(dataset): def generator(): for data in dataset: # Convert data from PyTorch tensors to TensorFlow tensors data = [tf.convert_to_tensor(x) for x in data] yield data # Create a TensorFlow Dataset from the generator dataset = tf.data.Dataset.from_generator(generator, output_types=data[0].dtype, output_shapes=data[0].shape) return dataset # Create a TensorFlow Dataset from the PyTorch Dataset dataset = pytorch_dataset_to_tensorflow_dataset(dataset) # Create a TensorFlow DataLoader from the TensorFlow Dataset dataloader = tf.data.DataLoader(dataset, batch_size=32, num_parallel_calls=tf.data.AUTOTUNE) image = tf.io.read_file(filename=filepath) image = tf.image.decode_jpeg(image, channels=3) #or decode_png The opposite of unsqueeze and squeeze is expand_dims : img = tf.expand_dims(img,axis=0) yield the desired/necessary transformations. As for the photos, I am quite sure that you missed a /255.0 in case of PyTorch or added a 255.0 division in case of TensorFlow. In fact, when digging deep into the Keras backend, you can see that when you call your preprocessing function, it will call this function here: def _preprocess_numpy_input(x, data_format, mode): \"\"\"Preprocesses a Numpy array encoding a batch of images. Arguments: x: Input array, 3D or 4D. data_format: Data format of the image array. mode: One of \"caffe\", \"tf\" or \"torch\". - caffe: will convert the images from RGB to BGR, then will zero-center each color channel with respect to the ImageNet dataset, without scaling. - tf: will scale pixels between -1 and 1, sample-wise. - torch: will scale pixels between 0 and 1 and then will normalize each channel with respect to the ImageNet dataset. Returns: Preprocessed Numpy array. \"\"\" if not issubclass(x.dtype.type, np.floating): x = x.astype(backend.floatx(), copy=False) if mode == 'tf': x /= 127.5 x -= 1. return x elif mode == 'torch': x /= 255. mean = [0.485, 0.456, 0.406] std = [0.229, 0.224, 0.225] else: if data_format == 'channels_first': # 'RGB'->'BGR' if x.ndim == 3: x = x[::-1, ...] else: x = x[:, ::-1, ...] else: # 'RGB'->'BGR' x = x[..., ::-1] mean = [103.939, 116.779, 123.68] std = None # Zero-center by mean pixel if data_format == 'channels_first': if x.ndim == 3: x[0, :, :] -= mean[0] x[1, :, :] -= mean[1] x[2, :, :] -= mean[2] if std is not None: x[0, :, :] /= std[0] x[1, :, :] /= std[1] x[2, :, :] /= std[2] else: x[:, 0, :, :] -= mean[0] x[:, 1, :, :] -= mean[1] x[:, 2, :, :] -= mean[2] if std is not None: x[:, 0, :, :] /= std[0] x[:, 1, :, :] /= std[1] x[:, 2, :, :] /= std[2] else: x[..., 0] -= mean[0] x[..., 1] -= mean[1] x[..., 2] -= mean[2] if std is not None: x[..., 0] /= std[0] x[..., 1] /= std[1] x[..., 2] /= std[2] return x mean and std mean = 0.0 std = 0.0 for images, _ in dl: batch_samples = images.size(0) # batch size (the last batch can have smaller size!) images = images.view(batch_samples, images.size(1), -1) mean += images.mean(2).sum(0) std += images.std(2).sum(0) mean /= len(dl.dataset) std /= len(dl.dataset) DataGenerator(keras.utils.Sequence): import numpy as np import keras class DataGenerator(keras.utils.Sequence): 'Generates data for Keras' def __init__(self, list_IDs, labels, batch_size=32, dim=(32,32,32), n_channels=1, n_classes=10, shuffle=True): 'Initialization' self.dim = dim self.batch_size = batch_size self.labels = labels self.list_IDs = list_IDs self.n_channels = n_channels self.n_classes = n_classes self.shuffle = shuffle self.on_epoch_end() def __len__(self): 'Denotes the number of batches per epoch' return int(np.floor(len(self.list_IDs) / self.batch_size)) def __getitem__(self, index): 'Generate one batch of data' # Generate indexes of the batch indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size] # Find list of IDs list_IDs_temp = [self.list_IDs[k] for k in indexes] # Generate data X, y = self.__data_generation(list_IDs_temp) return X, y def on_epoch_end(self): 'Updates indexes after each epoch' self.indexes = np.arange(len(self.list_IDs)) if self.shuffle == True: np.random.shuffle(self.indexes) def __data_generation(self, list_IDs_temp): 'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels) # Initialization X = np.empty((self.batch_size, *self.dim, self.n_channels)) y = np.empty((self.batch_size), dtype=int) # Generate data for i, ID in enumerate(list_IDs_temp): # Store sample X[i,] = np.load('data/' + ID + '.npy') # Store class y[i] = self.labels[ID] return X, keras.utils.to_categorical(y, num_classes=self.n_classes) class ImageDataGenerator(tf.keras.preprocessing.image.ImageDataGenerator): def __init__(self): super().__init__( rescale=1.0 / 255.0, rotation_range=10, width_shift_range=0.2, height_shift_range=0.2, zoom_range=[0.95, 1.05], shear_range=0.1, fill_mode=\"wrap\", horizontal_flip=True, vertical_flip=True, ) class Generator(object): def __init__(self, batch_size, name_x, name_y): data_f = None # h5py.File(open_directory, \"r\") self.x = data_f[name_x] self.y = data_f[name_y] if len(self.x.shape) == 4: self.shape_x = (None, self.x.shape[1], self.x.shape[2], self.x.shape[3]) if len(self.x.shape) == 3: self.shape_x = (None, self.x.shape[1], self.x.shape[2]) if len(self.y.shape) == 4: self.shape_y = (None, self.y.shape[1], self.y.shape[2], self.y.shape[3]) if len(self.y.shape) == 3: self.shape_y = (None, self.y.shape[1], self.y.shape[2]) self.num_samples = self.x.shape[0] self.batch_size = batch_size self.epoch_size = self.num_samples // self.batch_size + 1 * ( self.num_samples % self.batch_size != 0 ) self.pointer = 0 self.sample_nums = np.arange(0, self.num_samples) np.random.shuffle(self.sample_nums) def data_generator(self): for batch_num in range(self.epoch_size): x = [] y = [] for elem_num in range(self.batch_size): sample_num = self.sample_nums[self.pointer] x += [self.x[sample_num]] y += [self.y[sample_num]] self.pointer += 1 if self.pointer == self.num_samples: self.pointer = 0 np.random.shuffle(self.sample_nums) break x = np.array(x, dtype=np.float32) y = np.array(y, dtype=np.float32) yield x, y def get_dataset(self): dataset = tf.data.Dataset.from_generator( self.data_generator, output_signature=( tf.TensorSpec(shape=(), dtype=tf.int32), tf.RaggedTensorSpec(shape=(2, None), dtype=tf.int32), ), ) dataset = dataset.prefetch(1) return dataset def _load_image(self, image_path): image = cv2.imread(image_path) # BGR # image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # image = tf.io.read_file(image_path) # image = tf.io.decode_image( # image, # channels=self.num_channels, # dtype=tf.dtypes.uint8, # expand_animations=False, # ) # image = tf.image.resize( # image, # self.dim, # method=tf.image.ResizeMethod.BILINEAR, # preserve_aspect_ratio=True, # antialias=False, # name=None, # ) # if not issubclass(image.dtype.type, np.floating): image = image.astype(np.float32) # image = image.astype(tf.keras.backend.floatx(), copy=False) image = self.apply_image_transforms(image) # 'RGB'->'BGR' # image = image[..., ::-1] # image = tf.image.convert_image_dtype(image, dtype=tf.uint8, saturate=False) # image = tf.cast(image, tf.float32) # / 127.5 # image -= 1.0 mean = [103.939, 116.779, 123.68] # mean_tensor = tf.keras.backend.constant(-np.array(mean)) # if tf.keras.backend.dtype(image) != tf.keras.backend.dtype(mean_tensor): # image = tf.keras.backend.bias_add( # image, # tf.keras.backend.cast(mean_tensor, tf.keras.backend.dtype(image)), # data_format=\"channels_last\", # ) # else: # image = tf.keras.backend.bias_add(image, mean_tensor, \"channels_last\") # image[0, :, :] -= mean[0] # image[1, :, :] -= mean[1] # image[2, :, :] -= mean[2] image[..., 0] -= mean[0] image[..., 1] -= mean[1] image[..., 2] -= mean[2] # image = tf.keras.applications.vgg16.preprocess_input(image) \"\"\" Preprocessed numpy.array or a tf.Tensor with type float32. The images are converted from RGB to BGR, then each color channel is zero-centered with respect to the ImageNet dataset, without scaling. \"\"\" return image Unfreeze specific layers Here is one way to unfreeze specific layers. We pick the same model and some layers (e.g. block14_sepconv2 ). The purpose is to unfreeze these layers and make the rest of the layers freeze . from tensorflow import keras base_model = keras.applications.Xception( weights='imagenet', input_shape=(150,150,3), include_top=False ) # free all layer except the desired layers # which is in [ ... ] for layer in base_model.layers: if layer.name not in ['block14_sepconv2', 'block13_sepconv1']: layer.trainable = False if layer.trainable: print(layer.name) block14_sepconv2 block13_sepconv1 Compute the trainable and non-trainable variables. import tensorflow.keras.backend as K import numpy as np trainable_count = np.sum([K.count_params(w) \\ for w in base_model.trainable_weights]) non_trainable_count = np.sum([K.count_params(w) \\ for w in base_model.non_trainable_weights]) print('Total params: {:,}'.format(trainable_count + non_trainable_count)) print('Trainable params: {:,}'.format(trainable_count)) print('Non-trainable params: {:,}'.format(non_trainable_count)) Total params: 20,861,480 Trainable params: 3,696,088 Non-trainable params: 17,165,392 tensorflow-macos Releases tensorflow-macos tensorflow-metal macOS version Features v2.5 0.1.2 12.0+ Pluggable device v2.6 0.2.0 12.0+ Variable sequences for RNN layers v2.7 0.3.0 12.0+ Custom op support v2.8 0.4.0 12.0+ RNN performance improvements v2.9 0.5.0 12.1+ Distributed training","title":"Framework"},{"location":"framework/#framework","text":"","title":"Framework"},{"location":"framework/#pytorch-vs-tensorflow","text":"The image range is different for each framework. In PyTorch, the image range is 0-1 while TensorFlow uses a range from 0 to 255. To use TensorFlow, we have to adapt the image range.","title":"Pytorch vs Tensorflow"},{"location":"framework/#to-tf","text":"def dataset_to_tf( dataset, cols_to_retain, collate_fn, collate_fn_args, columns_to_np_types, output_signature, shuffle, batch_size, drop_remainder, ): \"\"\"Create a tf.data.Dataset from the underlying Dataset. This is a single-process method - the multiprocess equivalent is multiprocess_dataset_to_tf. Args: dataset (`Dataset`): Dataset to wrap with tf.data.Dataset. cols_to_retain (`List[str]`): Dataset column(s) to load in the tf.data.Dataset. It is acceptable to include column names that are created by the `collate_fn` and that do not exist in the original dataset. collate_fn(`Callable`): A function or callable object (such as a `DataCollator`) that will collate lists of samples into a batch. collate_fn_args (`Dict`): A `dict` of keyword arguments to be passed to the `collate_fn`. Can be empty. columns_to_np_types (`Dict[str, np.dtype]`): A `dict` mapping column names to numpy dtypes. output_signature (`Dict[str, tf.TensorSpec]`): A `dict` mapping column names to `tf.TensorSpec` objects. shuffle(`bool`): Shuffle the dataset order when loading. Recommended True for training, False for validation/evaluation. batch_size (`int`): Size of batches to load from the dataset. drop_remainder(`bool`, default `None`): Drop the last incomplete batch when loading. If not provided, defaults to the same setting as shuffle. Returns: `tf.data.Dataset` \"\"\" if config.TF_AVAILABLE: import tensorflow as tf else: raise ImportError(\"Called a Tensorflow-specific function but Tensorflow is not installed.\") getter_fn = partial( np_get_batch, dataset=dataset, cols_to_retain=cols_to_retain, collate_fn=collate_fn, collate_fn_args=collate_fn_args, columns_to_np_types=columns_to_np_types, return_dict=False, # TF expects numpy_function to return a list and will not accept a dict ) @tf.function(input_signature=[tf.TensorSpec(None, tf.int64)]) def fetch_function(indices): output = tf.numpy_function( getter_fn, inp=[indices], # This works because dictionaries always output in the same order Tout=[tf.dtypes.as_dtype(dtype) for dtype in columns_to_np_types.values()], ) return {key: output[i] for i, key in enumerate(columns_to_np_types.keys())} tf_dataset = tf.data.Dataset.from_tensor_slices(np.arange(len(dataset), dtype=np.int64)) if shuffle: tf_dataset = tf_dataset.shuffle(len(dataset)) tf_dataset = tf_dataset.batch(batch_size, drop_remainder=drop_remainder).map(fetch_function) def ensure_shapes(input_dict): return {key: tf.ensure_shape(val, output_signature[key].shape) for key, val in input_dict.items()} return tf_dataset.map(ensure_shapes)","title":"To TF"},{"location":"framework/#pytorch-dataset-to-tf-dataset","text":"import tensorflow as tf import torch # Assume that we have a PyTorch Dataset object called 'dataset' def pytorch_dataset_to_tensorflow_dataset(dataset): def generator(): for data in dataset: # Convert data from PyTorch tensors to TensorFlow tensors data = [tf.convert_to_tensor(x) for x in data] yield data # Create a TensorFlow Dataset from the generator dataset = tf.data.Dataset.from_generator(generator, output_types=data[0].dtype, output_shapes=data[0].shape) return dataset # Create a TensorFlow Dataset from the PyTorch Dataset dataset = pytorch_dataset_to_tensorflow_dataset(dataset) # Create a TensorFlow DataLoader from the TensorFlow Dataset dataloader = tf.data.DataLoader(dataset, batch_size=32, num_parallel_calls=tf.data.AUTOTUNE) image = tf.io.read_file(filename=filepath) image = tf.image.decode_jpeg(image, channels=3) #or decode_png The opposite of unsqueeze and squeeze is expand_dims : img = tf.expand_dims(img,axis=0) yield the desired/necessary transformations. As for the photos, I am quite sure that you missed a /255.0 in case of PyTorch or added a 255.0 division in case of TensorFlow. In fact, when digging deep into the Keras backend, you can see that when you call your preprocessing function, it will call this function here: def _preprocess_numpy_input(x, data_format, mode): \"\"\"Preprocesses a Numpy array encoding a batch of images. Arguments: x: Input array, 3D or 4D. data_format: Data format of the image array. mode: One of \"caffe\", \"tf\" or \"torch\". - caffe: will convert the images from RGB to BGR, then will zero-center each color channel with respect to the ImageNet dataset, without scaling. - tf: will scale pixels between -1 and 1, sample-wise. - torch: will scale pixels between 0 and 1 and then will normalize each channel with respect to the ImageNet dataset. Returns: Preprocessed Numpy array. \"\"\" if not issubclass(x.dtype.type, np.floating): x = x.astype(backend.floatx(), copy=False) if mode == 'tf': x /= 127.5 x -= 1. return x elif mode == 'torch': x /= 255. mean = [0.485, 0.456, 0.406] std = [0.229, 0.224, 0.225] else: if data_format == 'channels_first': # 'RGB'->'BGR' if x.ndim == 3: x = x[::-1, ...] else: x = x[:, ::-1, ...] else: # 'RGB'->'BGR' x = x[..., ::-1] mean = [103.939, 116.779, 123.68] std = None # Zero-center by mean pixel if data_format == 'channels_first': if x.ndim == 3: x[0, :, :] -= mean[0] x[1, :, :] -= mean[1] x[2, :, :] -= mean[2] if std is not None: x[0, :, :] /= std[0] x[1, :, :] /= std[1] x[2, :, :] /= std[2] else: x[:, 0, :, :] -= mean[0] x[:, 1, :, :] -= mean[1] x[:, 2, :, :] -= mean[2] if std is not None: x[:, 0, :, :] /= std[0] x[:, 1, :, :] /= std[1] x[:, 2, :, :] /= std[2] else: x[..., 0] -= mean[0] x[..., 1] -= mean[1] x[..., 2] -= mean[2] if std is not None: x[..., 0] /= std[0] x[..., 1] /= std[1] x[..., 2] /= std[2] return x","title":"pytorch dataset to tf dataset"},{"location":"framework/#mean-and-std","text":"mean = 0.0 std = 0.0 for images, _ in dl: batch_samples = images.size(0) # batch size (the last batch can have smaller size!) images = images.view(batch_samples, images.size(1), -1) mean += images.mean(2).sum(0) std += images.std(2).sum(0) mean /= len(dl.dataset) std /= len(dl.dataset)","title":"mean and std"},{"location":"framework/#datageneratorkerasutilssequence","text":"import numpy as np import keras class DataGenerator(keras.utils.Sequence): 'Generates data for Keras' def __init__(self, list_IDs, labels, batch_size=32, dim=(32,32,32), n_channels=1, n_classes=10, shuffle=True): 'Initialization' self.dim = dim self.batch_size = batch_size self.labels = labels self.list_IDs = list_IDs self.n_channels = n_channels self.n_classes = n_classes self.shuffle = shuffle self.on_epoch_end() def __len__(self): 'Denotes the number of batches per epoch' return int(np.floor(len(self.list_IDs) / self.batch_size)) def __getitem__(self, index): 'Generate one batch of data' # Generate indexes of the batch indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size] # Find list of IDs list_IDs_temp = [self.list_IDs[k] for k in indexes] # Generate data X, y = self.__data_generation(list_IDs_temp) return X, y def on_epoch_end(self): 'Updates indexes after each epoch' self.indexes = np.arange(len(self.list_IDs)) if self.shuffle == True: np.random.shuffle(self.indexes) def __data_generation(self, list_IDs_temp): 'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels) # Initialization X = np.empty((self.batch_size, *self.dim, self.n_channels)) y = np.empty((self.batch_size), dtype=int) # Generate data for i, ID in enumerate(list_IDs_temp): # Store sample X[i,] = np.load('data/' + ID + '.npy') # Store class y[i] = self.labels[ID] return X, keras.utils.to_categorical(y, num_classes=self.n_classes) class ImageDataGenerator(tf.keras.preprocessing.image.ImageDataGenerator): def __init__(self): super().__init__( rescale=1.0 / 255.0, rotation_range=10, width_shift_range=0.2, height_shift_range=0.2, zoom_range=[0.95, 1.05], shear_range=0.1, fill_mode=\"wrap\", horizontal_flip=True, vertical_flip=True, ) class Generator(object): def __init__(self, batch_size, name_x, name_y): data_f = None # h5py.File(open_directory, \"r\") self.x = data_f[name_x] self.y = data_f[name_y] if len(self.x.shape) == 4: self.shape_x = (None, self.x.shape[1], self.x.shape[2], self.x.shape[3]) if len(self.x.shape) == 3: self.shape_x = (None, self.x.shape[1], self.x.shape[2]) if len(self.y.shape) == 4: self.shape_y = (None, self.y.shape[1], self.y.shape[2], self.y.shape[3]) if len(self.y.shape) == 3: self.shape_y = (None, self.y.shape[1], self.y.shape[2]) self.num_samples = self.x.shape[0] self.batch_size = batch_size self.epoch_size = self.num_samples // self.batch_size + 1 * ( self.num_samples % self.batch_size != 0 ) self.pointer = 0 self.sample_nums = np.arange(0, self.num_samples) np.random.shuffle(self.sample_nums) def data_generator(self): for batch_num in range(self.epoch_size): x = [] y = [] for elem_num in range(self.batch_size): sample_num = self.sample_nums[self.pointer] x += [self.x[sample_num]] y += [self.y[sample_num]] self.pointer += 1 if self.pointer == self.num_samples: self.pointer = 0 np.random.shuffle(self.sample_nums) break x = np.array(x, dtype=np.float32) y = np.array(y, dtype=np.float32) yield x, y def get_dataset(self): dataset = tf.data.Dataset.from_generator( self.data_generator, output_signature=( tf.TensorSpec(shape=(), dtype=tf.int32), tf.RaggedTensorSpec(shape=(2, None), dtype=tf.int32), ), ) dataset = dataset.prefetch(1) return dataset def _load_image(self, image_path): image = cv2.imread(image_path) # BGR # image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # image = tf.io.read_file(image_path) # image = tf.io.decode_image( # image, # channels=self.num_channels, # dtype=tf.dtypes.uint8, # expand_animations=False, # ) # image = tf.image.resize( # image, # self.dim, # method=tf.image.ResizeMethod.BILINEAR, # preserve_aspect_ratio=True, # antialias=False, # name=None, # ) # if not issubclass(image.dtype.type, np.floating): image = image.astype(np.float32) # image = image.astype(tf.keras.backend.floatx(), copy=False) image = self.apply_image_transforms(image) # 'RGB'->'BGR' # image = image[..., ::-1] # image = tf.image.convert_image_dtype(image, dtype=tf.uint8, saturate=False) # image = tf.cast(image, tf.float32) # / 127.5 # image -= 1.0 mean = [103.939, 116.779, 123.68] # mean_tensor = tf.keras.backend.constant(-np.array(mean)) # if tf.keras.backend.dtype(image) != tf.keras.backend.dtype(mean_tensor): # image = tf.keras.backend.bias_add( # image, # tf.keras.backend.cast(mean_tensor, tf.keras.backend.dtype(image)), # data_format=\"channels_last\", # ) # else: # image = tf.keras.backend.bias_add(image, mean_tensor, \"channels_last\") # image[0, :, :] -= mean[0] # image[1, :, :] -= mean[1] # image[2, :, :] -= mean[2] image[..., 0] -= mean[0] image[..., 1] -= mean[1] image[..., 2] -= mean[2] # image = tf.keras.applications.vgg16.preprocess_input(image) \"\"\" Preprocessed numpy.array or a tf.Tensor with type float32. The images are converted from RGB to BGR, then each color channel is zero-centered with respect to the ImageNet dataset, without scaling. \"\"\" return image","title":"DataGenerator(keras.utils.Sequence):"},{"location":"framework/#unfreeze-specific-layers","text":"Here is one way to unfreeze specific layers. We pick the same model and some layers (e.g. block14_sepconv2 ). The purpose is to unfreeze these layers and make the rest of the layers freeze . from tensorflow import keras base_model = keras.applications.Xception( weights='imagenet', input_shape=(150,150,3), include_top=False ) # free all layer except the desired layers # which is in [ ... ] for layer in base_model.layers: if layer.name not in ['block14_sepconv2', 'block13_sepconv1']: layer.trainable = False if layer.trainable: print(layer.name) block14_sepconv2 block13_sepconv1","title":"Unfreeze specific layers"},{"location":"framework/#compute-the-trainable-and-non-trainable-variables","text":"import tensorflow.keras.backend as K import numpy as np trainable_count = np.sum([K.count_params(w) \\ for w in base_model.trainable_weights]) non_trainable_count = np.sum([K.count_params(w) \\ for w in base_model.non_trainable_weights]) print('Total params: {:,}'.format(trainable_count + non_trainable_count)) print('Trainable params: {:,}'.format(trainable_count)) print('Non-trainable params: {:,}'.format(non_trainable_count)) Total params: 20,861,480 Trainable params: 3,696,088 Non-trainable params: 17,165,392","title":"Compute the trainable and non-trainable variables."},{"location":"framework/#tensorflow-macos-releases","text":"tensorflow-macos tensorflow-metal macOS version Features v2.5 0.1.2 12.0+ Pluggable device v2.6 0.2.0 12.0+ Variable sequences for RNN layers v2.7 0.3.0 12.0+ Custom op support v2.8 0.4.0 12.0+ RNN performance improvements v2.9 0.5.0 12.1+ Distributed training","title":"tensorflow-macos Releases"},{"location":"git/","text":"GIT README.md \ud83d\udc4b Hi, I\u2019m @furyhawk \ud83d\udc40 I\u2019m interested in AI \ud83c\udf31 I\u2019m currently learning AI \ud83d\udc9e\ufe0f I\u2019m looking to collaborate on AI \ud83d\udceb How to reach me ... https://github.com/furyhawk git config --global user.name \"furyhawk\" git config --global user.email furyx@hotmail.com git config pull.rebase true commit msg : \u2502 \u2502 \u2502 \u2514\u2500\u2af8 Summary in present tense. Not capitalized. No period at the end. \u2502 \u2514\u2500\u2af8 Commit Type: build|cicd|docs|feat|fix|node|refactor|test Fetch/pull all branches git branch -r | grep -v '\\->' | sed \"s,\\x1B\\[[0-9;]*[a-zA-Z],,g\" | while read remote; do git branch --track \"${remote#origin/}\" \"$remote\"; done git fetch --all git pull --all How can I enable github notifications? install and authenticate with the github cli: pacman -S github-cli gh auth login ubuntu: type -p curl >/dev/null || (sudo apt update && sudo apt install curl -y) curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg \\ && sudo chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg \\ && echo \"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main\" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null \\ && sudo apt update \\ && sudo apt install gh -y reset to last working commit: git reset --hard git push --force working with submodules: git submodule update --init --recursive There are four steps involved when you delete a submodule. # 1. deinit the submodule git submodule deinit # 2. Remove the submodule directory from Git git rm # 3. Remove the submodule directory from .git/modules/ rm -rf .git/modules/ # 4. commit and push the changes # add submodule and define the master branch as the one you want to track git submodule add -b master [URL to Git repo] git submodule init","title":"git"},{"location":"git/#git","text":"","title":"GIT"},{"location":"git/#readmemd","text":"\ud83d\udc4b Hi, I\u2019m @furyhawk \ud83d\udc40 I\u2019m interested in AI \ud83c\udf31 I\u2019m currently learning AI \ud83d\udc9e\ufe0f I\u2019m looking to collaborate on AI \ud83d\udceb How to reach me ... https://github.com/furyhawk git config --global user.name \"furyhawk\" git config --global user.email furyx@hotmail.com git config pull.rebase true","title":"README.md"},{"location":"git/#commit-msg","text":": \u2502 \u2502 \u2502 \u2514\u2500\u2af8 Summary in present tense. Not capitalized. No period at the end. \u2502 \u2514\u2500\u2af8 Commit Type: build|cicd|docs|feat|fix|node|refactor|test","title":"commit msg"},{"location":"git/#fetchpull-all-branches","text":"git branch -r | grep -v '\\->' | sed \"s,\\x1B\\[[0-9;]*[a-zA-Z],,g\" | while read remote; do git branch --track \"${remote#origin/}\" \"$remote\"; done git fetch --all git pull --all","title":"Fetch/pull all branches"},{"location":"git/#how-can-i-enable-github-notifications","text":"install and authenticate with the github cli: pacman -S github-cli gh auth login ubuntu: type -p curl >/dev/null || (sudo apt update && sudo apt install curl -y) curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg \\ && sudo chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg \\ && echo \"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main\" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null \\ && sudo apt update \\ && sudo apt install gh -y reset to last working commit: git reset --hard git push --force working with submodules: git submodule update --init --recursive There are four steps involved when you delete a submodule. # 1. deinit the submodule git submodule deinit # 2. Remove the submodule directory from Git git rm # 3. Remove the submodule directory from .git/modules/ rm -rf .git/modules/ # 4. commit and push the changes # add submodule and define the master branch as the one you want to track git submodule add -b master [URL to Git repo] git submodule init","title":"How can I enable github notifications?"},{"location":"gpt/","text":"gpt https://github.com/furyhawk/nanoGPT train python train.py --dataset=shakespeare --n_layer=4 --n_head=4 --n_embd=64 --device=mps --compile=False --eval_iters=1 --block_size=64 --batch_size=16 logs iter 432061: loss 3.3774, time 102.72ms number of parameters: 3.42M No meta.pkl found, assuming GPT-2 encodings... The state of heaven with him yet have set The enam received in his power, It will be such,morrow; if he be By common enemies, so may not do, you As if you mark the truth. FLORIZEL: Why is my part; Not here my life, we'll do't, where it is The lark that the Earl of Warwick is slain: On bold, my rest, my liege, in whose good time I'll drown him worthy me to my Earl towards, Is Come to draw to scARD a fearful day. 3 KING HENRY VI KING LEWIS XI: What! canst thou live in poor soul, To do him banish'd at the same our point And the freshinks I come, 'tis aught to have but I fear you joys: all that I cannot On thee would of my heart, my poor soul, To be so bold a knave in this time; And thou by, she's children's blood, only, it is To see my life. MARIANA: A thousand part, indeed, sir, a time As you will wonder not: come, my liege, I well; And he is too a gentleman. See that with the voice that at once, you must Flried the senators of the any it good to us being son: but, I cannot be done; but we can tell him; for him I'll draw your words together. What you will know how you? COMINIUS: They are, uncle. CORIOLANUS: Therefore, my gracious friend, I am the farther un ches: What, poor! I beseech you, let it be a cousin. VOLUMNIA: Tush, the gods! MENENIUS: I am too! Nay begin to bid my lady play well: I am no pity that willy heir is right; But, by his leave I have or why it is. CORIOLANUS: Not? let me see, you are strange? They have made no more than all. Hath my wife's lord, upon pain of death. Third Citizen: Why, I shall have had some heard of you hear. QUEEN MARGARET: And so we all, I thank thee: 'tis more; Thou art the letter, and thou hast not so: wouldst be done, hark you, I know, my soul here have t something So to say, but I pray, and no more. ISABELLA: O, if it were full of such deep Norfolk, love, I would have more than his gracious fault Than Angelo first to you at great comfort. ANGELO: Yea, and make you he crook 'pt TrleKE: I'll make. Provost: What, my general? DUKE VINCENTIO: 'Tis like of his most, And leave me to the will for my poor side. ANGELO: Hath the king'ster heaven with the majesty of tears, With a kind: I have, my lord, must die. MARCIUS: Tell me hate them not, honesty are there To entreat of my wife, and my son to my order, And set the mother all my father's consent To make the must fly: so go: From whom that I would not, I do not fear, As well I will have me, you are vile toward. QUEEN MARGARET: But, by your vouchius, I fear not what I am: For since we came, your father's, at the least? Second Servant: Tis more than this of you can but have done My love or mercy to the queen of heaven: But I have forgot it, more, goodenio. CAPULET: Ay, as the matter I told you speak, my lord. BALTHASAR: But to the case of the Tower. ROMEO: Never; live, my lord, my soul is as well: I hope, by this which is a villain, When that thy brothers Romeo is, one sun With thy mother's wife, that I may return To know me what a dream hath as well As she run thought to be your mother, I I may not need to my sweet, CAPULET: Ay, what art thou? ROMEO: A queen, nurse? JULIET: Good lords, If thou dost know ye but. JULIET: Say, that Richmond's soul, take me in all. ROMEO: O, Romeo, I will not weep a while. JULIET: For the defry of grief makes them fair: But since unsapblful villain's wit of heaven It was a friend for ever-day, When I was lawful oath and not so, I am born to do it would it might be, I am revenged on-hearted my leaves dead king's oath, For the wantless blood whose heads must enter: 'Tis the Lord Northumberland that King Edward was himself, The Duke of Warwick, and his place in France, And thou shalt come to my will-day shall not stay. HENRY BOLINGBROKE: Why, howof, I beseech you talk of, Here must I play my heart with a cruelay; That doth not fall dead, I know the matter. DUKE VINCENTIO: You had even like'd, fie me, 'tis, I am resolved: Nay, unadBe now, I have heard her speak more than you I speak at peace. ANGELO: Well, I do beseech you, fair sir. Your eyes will not come upon me: he is most man, it is not so: so most much little most wrong, and may go to that by: which you have therein your hate, we may see themselves. LADY CAPULET: We are in prison, one word; ask him; For some you win me, upon his best: You know no better for the mind, there's no matter stands on him. DUCHESS OF YORK: How wouldst thou? Messenger: My gracious lord, I like youThis. DUKE OF YORK: Is it not dead, cousin. DUCHESS OF YORK: Then 'tis shame forAnd all that thou wast? KING R DUKE VINCENTIO: Romeo. Provost: It do trust me. ANGELO: Our lord, I cannot speak the good word: Since I havewas not it, I do'll therefore know Will a divineade me speak. ISABELLA: O, let me hear what you did deliver, By 'twere pity to him. This friends, be his head it, with one that's no matter, look to me as this lure of the prince a curse; more stands on't; No, sir; not a very weak of mine, If not that. Now, andso death's like death! I see, and hear Montague, my Montague. Who knows thou nothing; be not gentle, I may not, yet they shall be so, then; And had he come'd withAnd the oracle: I'll have your heart: that I shall follow that send him; that, we will warrant you, andPEY? ELBOW: Nay, he shall get a man to me. ELBOW: Look, very well, he shallEL' the better royal, which I do do't But I shall find, a time will be from his spirit. I prithee, take one: give me not your comfortier: I will not be come to-morrow; let's entreaty? ROMEO: I beseech your grace, sir,-- BISHOP OF CARke, my dTwere 't. BUCKINGHAM: I will, my lord: nay, You will have you go with me to please again. BUCKINGHAM: You will not hear duty, but for I will, sir, You have not the lady will with thee love But what you may! You are sure of he is to be a doth valiant; I will wot the prince: and have first to-morrow then. ANGELO: Thy business is the more and that my brother's sister. ISABELLA: It cannot be so. DUKE VINCENTIO: I say you, well; for if his present is the WARWICK: And thou wert enam'd; I'll not be mine own? KING LEWIS XI: What man is none than that at his house, That bear their down heart to Edward's-t thou him? WARWICK: Bid me, noble lord, our friends are fled, And sit to be true; and with all our sonsiss, From forth the most defend in theile departedath Be not born to make a husband as if I send it to thought that I did weep, And quench blood or not well. KING RICHARD II: O I was Clarence! What'sFL words were it: That I, as we are, that they have no friend, Even in the blood of heaven of fight, Thou p country's blood and'd. Come, and go with thee go; And, as mine I remember as any man's son As merry as I told the people and am mighty As'd an hour begin and bears our hands, But shame no other Paris and her be drawn. ROMEO: Is my long come. carry, let'st thou father In this remembrance of love: thou eat not 'em most noble swear tongue for what thou mayst, not wert between. The Earl of heaven of Warwick's love, this man, That's hearts, the true king, myMON and Sir. And unlook'd in the princess slain, Is beg of hen, and did yield in their arms At no moreOR create give from their hands, For the dead George of Clarence to his king, Who spake me his enemies shall hold it. BUCKINGHAM: My lord, my gracious lord, You had a power of wisdom cam thou out of mine? KING RICHARD II: He hath enem this, people, but I'll win me. CATESBY: Then here, my lord. KING RICHARD III: Nor I, the crown that queen is slain, To take the devil of our other linean ground, Not his our kingly curse people's-Which they that Is put to have open honesty Either to come in their arms. QUEEN MARGARET: ANTIGONUS: Hear you not? First Citizen: Come, sir; go, be it must none but you: I am ta'en and leave of you; and, then, go with me. First Servingman: Why, we will, sir. Second Servingman: I would not, no more: the provost. Third Servingman: What's the matter? Third Servingman: What's the matter? Second Servingman: No, my master, I can tell; I know how it you, thereof he m it. Clown: He's here; we should be a witness to the purpose. Third Citizen: He hath done what you are done. Third Servingman: A Romeo, sir, for them he would have found baw Rome of the world. CORIOLANUS: But I love the child, Not that thou hast other of thy life. Third Servingman: If thou she were thrice a man, And had these griefing force royal royal queen; And I am son, and I love thee myself. KING RICHARD III: But I will be poor; which he is he made? BUCKINGHAM: My gracious lord, let's see your grace be company. KING RICHARD III: Why, what a fellow should be? BUCKINGHAM: My lord, 'tis a subject, proud Which else, which, had some him, it will with heaven Till wind manage our doth made over his land. Why, proud I have ta'en the service of the people, That would not have more better that, Which will be satisfied; and, how his ignorant shouldER' He seems not? and I have had rather thy, To do my country, to make the'd right, With all the envy of this loyal, Our holy and be England and Duke of York, Not in his second soul I and have been The which never bid us, and call King Richard's lord. HENRY BOLINGBROKE: Welcome, uncle; we shall not be unto him; But so, my good son, I I do further. KING RICHARD III: Why, then, I confess thee, lord, If thou hadst never yet thou yet hadst never, But yet thou wilt wert up in thy horse. CLIFFORD: Clarence and Gloucester, I will not bethink, But that's not yet did nothing: but the gods The times of revenge! RICHARD: Nay, bring me what ancient, and this hand.' KING HENRY VI: But, to thee speak commonth very day! CLIFFORD: That thou wert so disgraced me to me. KING HENRY VI: Woth he the oath that I should tread upon our way? CLIFFORD: What may your grace in poor Henry's life and him? WARWICK: No, like a bawd, you not so; and, and for the poor one did, he is even to give his l. WARWICK: Uncle, so: I do intend to him in all, I hope. KING HENRY VI: How far I mean? if this be so bold with! The which, his queen, the queen, his love; And so, gentle king, may they do you in at night And fly him in the slander of his king, To be revenged on him that, so should you have; For, by this way be you found you to, She's a woman. ANGELO: He will not see you, sir. ESCALUS: I am aTo your request in hand, you shall in arms You must give, and yet go by, a course it is thence. ESCALUS: LARTI not mock mine own good. ANGELO: Do you hear. The duke hath forgot you for what you are? ESCALUS: BUS: for whose offence? CORIOLANUS: What to you? CORIOLANUS: O, worthy madam, And have I too most, you shall not. AUFIDIUS: Worthy sir, farewell. VOL Had not himself, nor known she! Provost: An me, both that; he did, sir, to steal-on Her mother's Romeo! O old faithful friar! JULIET: Why do I more than that I should never speak of. ROMEO: Nay, rather give me leave me that I will; For I shall find some better happy days Than lay, a man. FRIAR LAURENCE: But slain, her hath the next day a little. ROMEO: Nay, good woman; my turn in thy sake art full nights here. Who is dead and young prince. Ah, how, dost thou find me to thy breast, Thine hast he, and more, a noble fawd. Sir, thou canst not speak, I wouldst thou wert For sleep the mark of thy deadly years? Ah, keep'st, asLord men, thou hast got By so thy voice: there's no power, it is none honest. MERCUTIO: Tybalt, Romeo, whom I have the best I think how I have done. ROMEO: Thy life art thou, that thou art too fair! BENVOLIO: Mehe, and go not in 't. MERCUTIO: Nay, I'll bring some noise for a for that. CLAUDIO: Your face? aarer is a traitor's head to pluck him to the king the house. But which you can do I have to the hand; for it will be, as I am and a block withFor a man that Claudio hath married, If be a uncle. DUKE VINCENTIO: He did know more. You are most little more, he hath done it well, our the city and do them to make me theD:' Aufidius come to me and brother; and she isome with those that hath been, 'twere pity us all. Then, soft, wot each that am aly; For now I did well know. My mother! LEONTES: What's well man? CAMILLO: KING RICHARD III: Be not so? HENRY BOLINGBROKE: So that of God he would have the king his! BUCKINGHAM: I'll make more than my gracious case: I will bear the deed. KING RICHARD III: tis you by king, I would am I bethink, To make an amissant in love That thou wouldst protest to keep post toOR'n; And thou he is not nowSo. LEONTES: Unman, it is; Nay, good my good lord, be satisfied still. LEONTES: A most business: Let me have been since I was to be mercy; Whiles the lark-f orth-she, He cannot, Marcius worthyKE: 'tis well stock no poison. The manner of the envy he spake to Marcius, That with the power is worth dead, the doth be brief, In worthy Romeo hastous tune up. KING RICHARD II: Whom was it so, my child is set on death. HENRY BOLINGBROKE: ClOR, my lord; With late that cons news, for, to that I would Trueass the great number pardon'd, from your head already're his mind i' the there, I will. DUKE VINCENTIO: You do but see your highness of your arms Would know the king's mouth with peace. MARIANA: I mean, a son, it is not a word through a tear for Claudio, and be banished; But when you bid this manner bless us, In whom we have had been wont to do, 'twas upon the business, you I shall call'd his friends to him, and then. DUKE OF Lord: He's tearsark; for long my gentleness' I let me hear; good cousin, adieu! Belike his hands I in his remedy; And I have died to face, my love to me, And I will make him leave of can say, I had rather keep their words with him to keep him Whither, to thy fortune but put off; And see that this terror may be so, So if that cannot: so that's the friend, That will be deservedly of your honour's life, Let thy rage here cut in his grave; One side do not. MENENIUS: Hath the people's great state by power, That they shall have set up a part; But, as minetis the nothing of a man, That is the deputy,tis well for fault, And breathed his honour with the fear of death, That all for that as you are, and we may live. BALTHASAR: A pretty fardman! RICHMOND: Bid him be here? and I'll make me pardon up. RIVERS: What, hast he not? BUCKINGHAM: My lord, my gracious lord. KING RICHARD III: As I remember; against thee, thou wert not so much done, That thou canst swear and thee, for thou hast no cause. RATCLIFF: I will not be not, my lord. KING RICHARD III: Ay, O my children, my eyes! BUCKINGHAM: No, mighty lord, I hope, shall understand it. KING RICHARD III: I know no? but himself he is done; The side must be seen and make me wrong. QUEEN ELIZABETH: That thou mean I sent to see thy life To have no more fit than it is in this world To look upon thy brother's make me hear, The Duke of Norfolk; if thou darest with Thy Lord Northumberland, rouse'd and thy crown, For thou shalt no life but by thy kingdom. KING EDWARD IV: But thou, in my turn, my grave jest thou, You, the newats, and have the ears, Where fruit men have done-bended, then my daughter Of happyness! O my woman! If thou didst, thou tis'st fair a piteous, which he was, And know not 't prove a thousand right while you That you are to have something: so, my kingdom, 'Tis just that I iter 506998: loss 3.5516, time 71.91ms iter 506999: loss 3.6803, time 72.24ms iter 507000: loss 3.3026, time 72.24ms iter 507001: loss 3.4185, time 71.62ms iter 507002: loss 3.6271, time 73.12ms iter 507003: loss 3.5836, time 71.73ms iter 507004: loss 3.1981, time 75.92ms iter 507005: loss 3.6141, time 71.98ms iter 507006: loss 3.2294, time 73.99ms number of parameters: 3.42M No meta.pkl found, assuming GPT-2 encodings... The state of heaven with him, and that his body is humray'd this. WARWICK: And I. DUCHESS OF YORK: My dear lord, DUCHESS OF YORK: HENRY BOLINGBROKE: As far go we as any we toOL? HORTENSIO: I think she is well: then will you have so: I would we were not been then in war's good. DUKE VINCENTIO: We shall not be still be gone and been to see. GREMIO: And your good lord, I would all and like you; and in your hope. ISABELLA: I can know the truth, I am your father's house. DUKE VINCENTIO: It is no gone, sir? LUCIO: I think I would, sir, I know say, I am a friendlike by him: it is the very much of your blood; if Would I be well show'd of you in the bosom And our new-a. Provost: MAMILLIUS: My lord, I'll warrant thee a holy heart: which's with mine, I am too good to him. LEONTES: I am none; Sir, for your son I have here in this business To the justice of your father's worth. PAULINA: It is my gr in that I were, To the most of your own: your therefore have Be consul: Come, come, go, go, be much here too much: I am no more. LEONTES: They are such a and the and yet. HERMIONE: The better, sir; I am an't that you have notity. LEONTES: LEONTES: Well, well, what! LARTIUS: I am so. PAULINA: I pray any I, sir? PAULINA: Alas, poor woman! What ha HENRYHAM: Too true Clarence was a holy bed. CAMILLO I shall have had a brother of your son, To yield your brother's brother. LordOUCESTER: CLARENCE: KING EDWARD IV: KING EDWARD IV: Why, then, think you what: go-morrow, son, That you must not do that I am just, For when you can, my lord, my brother, Henry With your brother, if he bring end to your last. And, high-ed, if you see her you do, He would not have been thus an whose speak With that most un Butice: I would, he were born, a name: The house of him I'll speak to see your wife's. DUKE VINCENTIO: 'Tis not her in, sir, his friends, which Of wrongstAName to the earth, with a little way. ISABELLA: I am so mither'd by the point of it That I have had. DUKE VINCENTIO: O, let me think; and let me see a word. Provost: I wouldISABELLA: Wilt thou not, then, not in a very kind for a very for me that I did use me from him to speak. And I beseech your majesty and your bosom And my great opinion, have a wife and father, Unto my brother father and my well. FRIAR LAURENCE: And there is my hNor talk'd be no sworn to death But that I should think 'twack the part of you In the duke's daughter. Nurse: A love a man that his blood is more a life Than this may standow make her false friends. Nay, more farewell; I not for't; Our unf shors'll ask you: come by the man-likeent of this lady is the least, But to the everatter of me dearest day, Or me but one earth, like a thing that Not so much sorrow's, as many more is true Than this in his own! as our bthe light As I do! What's in all my shame already may in openings! You, my sweet HASTINGS: Be early what would you say to our lord? YORK: It is not so. GLOUCESTER: It is. LADY ANNE: sweet my old grace, that thou follow'st! GLOUCESTER: And, for God's sake, till you are three years. LADY ANNE: What, prince? GLOUCESTER: No noble thing I should be a happy king? GLOUCESTER: Away with him! DUCHESS OF YORK: My lord, I must not need: Yere to my children's love's loss of thee, But thou, even to the king, thou thy doubt'st tongue. KING RICHARD III: What is't o'So? DUCHESS OF YORK: I, by my life, I have done, but so in once For in the virtue of our blood: without that kind Was never so far gGLOUCESTR well, And I will follow him. DUKE VINCENTIO: M King Richard, give me leave to my thou thou go Be at home of these good fairWARD. But of what, thou dost to do Edward! B vouch'd: My lords, more than you shall please our royal name. LORD ROMASET: So, to save the king; yet he's just, And come, to answer me in him my country, And you shall have you where we did say the prince? DUCHESS OF YORK: My Lord, may you do this dangerous lady, My father's heart, which your hand did thee yet be The first one of this noble Edward's king; And yet that is no more: come to thee, For I am we of my your love. KATHARINA: I have a very. KING RICHARD III: sadful lord, and his part-f ready I come To make the king a king from him there is That he's made good. Come, let you go; But there be 'twere noIC at you all; But you shall have France, for honour; if you were now have My body hath done to banish yourself, To prove a good will-dew heart, Wherein my noble prince was comfort To make a man's name and a great bed: 'Tis well aKINGBe, he's in a child I'll be by my husband; let me she: I'll have it if I use my sake. Why, how, let is in that sword this traitor! HENRY BOLINGBROKE: Richard, help! but more than his that I'll make my tongue to love thee such life As thou art to Romeo. KING EDWARD IV: This? but doth he not Edward's death's death! HEN MARGARET: It is more than no more yet than by it. BUCKINGHAM: What, is it so? GLOUCESTER: I shall be there was in my heart With gracious lord and my good lordle brought He is well: but I hear you will Give me to my duty, to the world. First Murderer: Because thou canst love, that ever, thy love, thy child, Thy man shall take a grave on thy wretcheding. Fare not thy sword, content Camillo, give Of your she on a my master and son; And here, I beseech your highness,-- FRIAR LAURENCE: eyeTYou may my kindred again! ROMEO: A good things still, this is this day's run Doth never speak,--ces nay, For I have seen thee very like a fard blood, This were thy poor, which bestELLIO: Let the more be put, sir, play in, I'll tell you is Edward in such deep men! KING EDWARD IV: By heaven, I hear no more: my brother's heart. HEN ELIZABETH: I am come to have a more worth in this. KING RICHARD III: Allak thou, God! it is not my son. QUEEN ELIZABETH: There is no more I swear than they shall grief is so. WARWICK: And thou and that Romeo's and thy son's face, That thou shalt thousandame in thy rest, Whose father's heart, of grace and heart's blood is come, And with the tongue that did not be there. BENVOLIO: T Gm would cause, I wouldsthip wish it true. ROMEO: Is thy life- vouch's own word. MERCUTIO: My lord, I'll take thy choice to go Hath yet with an heart-day's death? BENVOLIO: I dare not for the best upon thyPle I talk of thy Ver lives. Come, let's not A noble man: For she hath yet a man to fall? ROMEO: Thou art not so well:OM pity her: But he's an but one. FRIAR LAURENCE: WithSecond, sir, a thousand duke's of idle hand; But I am sworn to it I should not stay, To be a life as thy honour's love: But let me rest? JULIET: This will be, to put forth; there is no man. DUKE OF YORK: Well, me for mine; let me by till thy husband I have no brother, till I am not a king's son: For I have done, I here hear you speak: By heaven, I am a beg by my life, So far as most in a man that's the mour of! O, that I would be less! the great which o' the honour, And that I had; but then, I am a-beLEath slain, Were kill my takest! If never, run Aufidius, 'Tis thought it would you are at enmity That you did but so much is, and And I'll lay down out whose't be fullle In such most I have kill'd his have been such his, As so in the world is dead. Messenger: The man that he is our lady. KING RICHARD III: He kill, my lord; who, we could not say 'ld have so much. Come, come away. Of God's name and our fair? 'Tis oath? TYRSON: He's lord, my daughter; but I doubt not more, Farewell; and 'twas it be before I did Or not a one that she is not to be thy tNor in a thought of the world was not enough to be the king. Provost: DUKE VINCENTIO: How? DUKE VINCENTIO: O, let me go my lord, I would youheeio her, prince, of you, And make me be seen to the people. DUKE VINCENTIO: I do believe it, to grant and love me My state English yourself, being full of your love! O prince! one of you a man that I must not all be the king, which is mine and him. LUCIO: But, madam, an'tess'd by the do break your honour. DUKE OF YORK: I, for an Murd-- KING RICHARD III: What! what goes this? at' lie I'force? DUCHESS OF YORK: I would I had a poor day for- thou art made it. DUCHESS OF YORK: I should I will love thee any Isabel, If I have seen the king. Provost: DUKE VINCENTIO: Good Romeo, Romeo: 'twas done, As that in any man have done to you wretch with him: So, well I desire it, to my heart Is this the day of Edward's wife: Thou most Richard, I fear me, Thou wilt make thee as thou wilt be a thousand As I am, as most gentle for ourselves: To this my husband's fair day, my soul, I therefore let her stay with heavy kiss: And, as I could, but I would, my good lords, Ere you be king in oath, and my death. LUCIO: Why, how thou hast done this man with thy love! Farewell! thou dead too, what 'twas no time Thy other blood should come; and with that WhosePOL know to fain: 'Tis so false, to beg mercy, their pluck'dine side For their goodILL'er's hands. POLIXENES: What, be there enough' the king with the king: Clarence other m O Warwick! if it be done, And that my good cousin, I wot that I know. LEONTES: How can you think! We must be this be you; this power are My true son, my brother. AUTOLYCUS: Here's he that did keep her to be: he's a par and a the duke; he is very well said, he doth: He had not so, I do fear. AUTOLYCUS: I may say, sir, my queen, dear love. Clown: My lord, you will, by my life, a word; the he duke's daughter's wife! MAMILLIUS: I have not been i' the lie. DORCAS: I know thy canre so. POLIXENES: O my lord! MAMILLIUS: No, my good Paulina,-- DORCaius out,-- DORCLEY: So much to it, sir, that it may be received To the sons; your pleasure where, the king's heir, Have he been nothing to be depose, But the poor law so far the rest is dead. KING RICHARD III: Ay, if I have Clarence's bosom forth. QUEEN ELIZABETH: WhyWARD, Well, pray, go with us? he is very comfort. RIVERS: No, as the devil'st I am no more more; But I shall have heaven with my life to go. QUEEN ELIZABETH: How long the justice of the earth is not mine? GRUMIO: prison does: Come, sir, it is no more. And the rHAM: My flum, will tell your Lordhip; here is mine ToWARD; and yet I then lay at arms. YORK: God save your lordship to his own Lord Hastings! I but on such right and said 'tis well. KING HENRY VI: O, then, I'll away thy life to bear. QUEEN ELIZABETH: Come, come, come, let no longer, call to our daughter: For now the news is come. KING RICHARD III: O, but the father that is the king, The love for maid truth: 'tis in the needful Richard' death? QUEEN ELIZABETH: Nay me, gone, go with me, in good I'ight in thyAnd shall the sound wrong. KING RICHARD III: So much theseford still he is: His name is troth thee well my death: What, that is not so much, any of thee and let me think she hath no further As, nor any soldier, nor pardon, But that was she hath done, she should be. STANLEY: It is no more of your bed hath best not true. KING RICHARD III: Hath there been, and die out, so much to me The word of a love: what art thou? Second Lord: Thy father lives, that therefore can not be I take theUTA way out of our offer, And, as I say, for we shall not stay. LADY CAPULET: O my hand, a fault will be more never heard Or else moreare. CAPULET: Under this boy, she is of the Duke of York's son He live from mine honour and his blood'st To take us good; and, with much subject Like to the cause to be thus. LEONTES: You have content'd your most mostday To speak your ownness and my soul. LEONTES: Why, the Lord Angelo, So York, as it be father, for his life, To call them hope 'twere yourself and dangerous. CAMILLO: Why, what a thing? LEONTES: Was everBut in such a time of her, Or in a good will of this unisest? KING EDWARD IV: What, lords? WARWICK: And Warwick Clarence at the eyes of justice! KING EDWARD IV: Why,ere no more. CLARENCE: GLOUCESTER: CLARENCE: HASTINGS: KING RICHARD III: Death, brother, are you through my soul Sray to me, and leave me to my uncle? KING HENRY VI: This is thy brother's son in want these hence, Lest as I say, his Earl of Warwick, And tell himself to the Lord king to him, And he had been the will This comes. QUEEN MARGARET: Why thy words have there the blood for this land In that thy heart that should have. You, God's right I should have needful nothing set upon, Theself I am: and yet thy doth-- QEN ELIZABETH: O nothing, that my lord Richard will have. KING RICHARDISET: Yet, as I come! KING HENRY VI: I know the name do march. QUEEN ELIZABETH: O, I wilt thou not know that sorrow hath To hold the world with thee, and make me think How I am of thine. KING RICHARD III: In whose from all the shame of save, ThatCORIOLANUS: Why, how's there,-- VOLUMNIA: Let's hear your heart? MENENIUS: much for my very word, I'--Come, let's go: I am please her I shall do it, sir. MENENIUS: It is a cause, Pray you, get your words. BRUTUS: What, our good lord? SICINIUS: He shall, you shall be; he are three gods Is full voice. First Senator: MENENIUS: Ay, but I will go. First Senator: I know your Rome. BRUTUS: Is the news prove true, my lord, and thySo, as I And see that this beable and mostadman: Hath not a high Mercutio? That's more to say 'twere well;' which I pray I am too more than some, but so good For one to be done. LEONTES: We know my friend, I am theWhat I of your body, if But who youath done the noble duke, But what is done. LEONTES: Good Marcius, you have lost thy devil, to lie all Good night. POLIUS: The people did. SICINIUS: Are you all Aufidius? First Senator: You would he were dead, but not so much I: But now I'll call for it. MENENIUS: Well, well, no more. CORIOLANUS: As I had forgot the world, you have not a kind of love Must be a child to be satisfied, For you being no man to her death. BRUTUS: Away! the arm Of your fair wife'st, For she was my enemy. What doth the man! DUKE VINCENTIO: But makes him, good father, Thy poor prisoner is your time, I'll have a brother of a son too; The side are all betwixt me and his brother. And I have need not; and, as I am not The goodly gentleman is too, do not But else 'INCENTIO: O,en O, good me, indeed and I'll thank thee not no He hath deservedUKEill live to do this; And, if this beNORine no man's death is cold. AUTOLYCUS: Away! he is not made good, sir. DUKE VINCENTIO: My heart, my good lord, did't not't. ANTIGONUS: It is of this; We know of you are now. Servant: My father is at hand, Who came Camillo: 'Tis goodly: I'll be the bosom on the world: When the common misfoler of this battle, To iter 599988: loss 2.8334, time 71.35ms iter 599989: loss 3.3947, time 71.09ms iter 599990: loss 3.6317, time 72.32ms iter 599991: loss 3.5352, time 74.15ms iter 599992: loss 3.2102, time 72.34ms iter 599993: loss 3.5483, time 71.71ms iter 599994: loss 3.5832, time 71.76ms iter 599995: loss 3.4675, time 71.01ms iter 599996: loss 3.3202, time 71.27ms iter 599997: loss 3.7681, time 71.95ms iter 599998: loss 3.6641, time 74.20ms iter 599999: loss 3.8146, time 71.89ms step 600000: train loss 3.6565, val loss 5.9145 saving checkpoint to out iter 600000: loss 3.3908, time 193.50ms number of parameters: 3.42M No meta.pkl found, assuming GPT-2 encodings... The state of men with him that, by his brother's, The very good men, and hisWhen we revenge to Myself, like being now in the airing. GLOUCESTER: This gentleman is set out of other. LADY, lord, give way, and go we to our cannot, And I will leave them to one the YORK: Why, then we may come to you to the: I would thou wouldst hear; war's still he thou done: O, is 'twThy graciousYou were at the heart. First ay: I did not stay? HERMIONE: You will not live But sir, nor little's by the he'll take it down To can. You are not fit to me? LEONTES: What, youPEY? MELMy words may bear't. The one would have so. ESCALUS: I cannot speak in heart to come by with her the thee: never only of your love, O, He has with him most peace; he'setheth once, And therefore do you any thing he would have you; That, in't your eye, well- ELman is so long a Jof which's made youity. If you, you may hear that you are, wtis a shepherd only may he say Bohemia. DUKE VINCENTIO: To your love. LUCIO: ANGELO: I am that king. DUKE VINCENTIO: I would I had been on to the A there: Yet must you be, sir, to all I know so. DUKE VINCENTIO: The duke shall know you say speed again to yourself you. Provost: I will be an't. DUKE VINCENTIO: It is a word of me. ANGELO: TISTRESS OVERDONE: Ah, howio! not all as he being As is a father of a e'er these adTan your brother is born. MAMILLIUS: I am a word to-morrow; youpt, But you might, I shall have had some to him thus. LADY CAPULET: If you were mad? 'Tis full of what you should have. CAPULET: What, say'st that I mean to me to thy name? JULIET: I am the more. Nurse: Is a word. JULIET: What is the news thou bring? ROMEO: How shallest what's a d see of them! ROMEO: O, give mewence, that comes in the time Of this. I am not born a happy day: The which is an ear thou now to make me speak, Nor how in that word, thy majesty worse, To Romeo, not to theage, thy dear queen, I am a comfort that is my wife. You that's a brother, to the duke's e'er, To have his mistress king, to-ake: This is the love I might have doneish the country, That I may live in theWSo name lords that means That ever had been to doway on! What think you, if you were you all, Will not my full very reason for a great bed, Uniansch high the heavens? answer this Of this fair prince's name? CAMILLO: I will, my lord. LEONTES: Pray your father! That's true-Sir, and you are son. POLIXENES: You are rough sir, give me Lord: let him go your seek. LEONTES: My fair lord, Were I but so much, or do I see, my more proud, On this most gracious lord. ANGELO: Thy great youth is the forth knows. ISABELLA: But 't, an confp Henry!' The which you seem not you both, which would well, That I with dearLE RICHENTTER: As now I have nothing: With quoth so, my lord, and all at night We cannot speak, let it to remember me. CLIFFENCE: Is some thought of this, we may be a cause. KING EDWARD IV LADY CAPULET: And to me have me bawd, youngly, I would say, If you say so, it shall know the world thou thyself. LADY ANNE: And, I am that day to last with a thine. JULIET: O God's sake, is come! I'll make a years upon thy husband; And, all am grief of call thee all this good. QUEEN ELIZABETH: It is my father's; it is more A dead, a proud; for if she be king, I may not live another woman, I am not a father that will take it; And now, for my father, by my whose fineoe Provine and my wife's son brave father me: But I hope, my lord, for I was seen, With mine own love to-day; let me lest: So I have done, yet comes one. DUKE OF YORK: Thy kind-morrow spake for the grace-night! DUKE OF YORK: ay Richard, that dothst thou do to love thee? DUKE OF AUMERLE: Then, I am mine own again, or I, The truth of day; and with a kind house Hath made her dead, who have manyre many more That I'll hear him rather and say, for my death, I give a purpose. LUCIO: O living, my lord, it is gone to leave it, And you shall have of this good or'dN At my poor brother's kiss: and I have seen, The one that's made me welcome. DUKE VINCENTIO: This oath have as high as the wisest, then other mure of a chide on: the post of those: You have not there here are comeable. LUCIO: But in this while I should have them; if I cannot, I do put the fool: then, like I come to love thee With a set upon my master, which is his pence of the king; even so in his hands, As I havech you at his best, By that it may France, for honour! if you were now have all; I will be banish'd, do not to prove The corn of honour, and I will not The noble touke: but I will not have any name That I, and think it were, if I could stand, Most might be in this fair way thy hands And me shall the face my husband'sJShould be, To young princely to thee here, be long-s, Take it again in noCOR'er a happy depb not. ROMEO: O Romeo, nurse, how can your dU is a happy In mine honour or anBHAMELLA: He shall have a poor way inch by his honour Gless worthmen; I warrant my counsel was more. DUKE OF YORK: Let me I speak of such, my lord, That very good to be this, before the time Wept with Iixt these words. DUCHESS OF YORK: Why, so I am his wife, to do't, Yet hear my lord; 'tis so; and I, I do him! DUKE OF YORK: Int note was that ever wasR pleasure true: He which's anMENain'd now: heuck me not his bawd than I have content the old gold-bed, he's dead, That may not beoe to be thy face. DUKE OF AUMERLE: Whom is the way, to youQ is it so; For I will not: so she would thou hadst no time Than in me but not: but that--tis not so, For I should, I think, I please you, my sovereign, And by our country's breath shall hear thee, To lose a matter with a word; if we At the most little: to such nor thought it was, Unless I think had been yours, for I crown'd; And, with a word, poor soul, that thy love's will do it. BUCKINGHAM: I know the gods. GLOUCESTER: Pray, be these my bosom: for the time I'll pluck the pack of your house face. KING HENRY VI: WARWICK: And Warwick's death will go and did to us. And yet not be fear'd in all. Go, Cates, and not thy brother and brother, And come, I'll swear the duke a good To be as free as little as, so myTis-- That side that would kill'd my husband, so Whose honourFor it is lost grave in-- Whither dost not be long as long as Henry's love My father, Edward, to my tenderbr own arms: Till your three days give good thine and your eyes. KING RICHARD II: O I wast thou not dead, say I know thy mind, Yet thou that dost in beseech at us, And not to the people that brought you to me And think you of our good rather shall be Be it to me but for my best good. First Senator: O Clifford, O, it is the more! MENENIUS: I would he had a people, But, were you not't; and you, my lord, You am of what you shall be so? BRUTUS: When you are should have bound 'em. MENENIUS: So, most it do. BRUTUS: Where is the love of this fellow, that must not? AUFIDIUS: We do not say: We were to do't. CORIOLANUS: You have been a cause to need. VOLUMNIA: O, indeed, I'll to desire o'er a counsel, And I will play the us. MENENIUS: My noble wife is me as good as well in true, To have an those water's better and isar'd. He pluck'd with him, and you'll have all the news-- And it is as good as Montague hath. Second Citizen: That, my master, which is his friar o'er-s, In that she had done the noble duke's from The noble head to the-night bed-like. First Citizen: Sond to the king: But I hear, madam, let him be heard it Of God's name, that is ill and great blood On thy be these; yet every w thought I with it, And none but that I doubt; and, in your love Is it right: but I, my poor dear cousin, I hope that I had not to this brother, But I will stay in the thing I say. DUKE OF AUMERLE: Why, lords, what aunes may? DUKE OF YORK: A gage, prince wouldst thou tell thee they! Thy brother's death! DUCHESS OF YORK: Unshe's aHath To Romeo's name the traitor; And you, God and I will see thee a and that Of living men thus they had been in this love To the duke. QUEEN: So, they are not to be the name of all. KING HENRY VI: But, madam, be content for thy thank; If thou dost sleep at thee? Now long rest is dead; thou wert not thy f after? QUEEN MARGARY: As thou as now I love'd myself of thee? QUEEN: But I shall fight with him. KING RICHARD II: O God's pity where long the o'er the world-and'd ! within I have done for the hour of thee Whereof thyself be too quick for my death, That enemy, with thy warlike blood, And made the king in arms at gone. DUCHESS OF YORK: So shall I do, to so late, Why, proud-hINCENT are lived told? DUCHESS OF YORK: No? KING EDWARD IV: I know you that is a father, to my? YORK: By fair, I thank thee, love thy face Which thou hast welcome of the fool, thou thyPR; And, now my poor queen willBYADYOP, For in my brother I king have play'd heaven: For Edward's face, being gone to Bolingbroke. TheAU's aH no other tongue to thee; For I have sent not to be of. DUKE V WhosePOLIXENBERTER: Not that that shall tell thee to my grief, But I will not be gone to thee, and thou a present tongue. 3 KING HENRY VI CLARerer: Take it with the day. BUCKINGHAM: Therefore, my lord, I let not beauteousors. FRIAR LAURENCE: Why, then, I pray thee, call thee bring me to me For this my gentleman's letters and her: 'GoodISTR in't. ROMEO: You must beL blood; And thou wert dead with a for him thou this! Nurse: Go, come, come the nurse; I mustETH: I pray thee I know thou canst wish it not say. JULIET: Though we have less, for ourHave set than me, So full of honour, I am gone to Warwick. ? of ourmen born? Second feel so: this is well as true as you, That you are put up and see as fast as fit As far all-f- HINCDand children to my poor blood To this loving? LUCIO: Doth she, when her father's Claudio hath done. What's he? He is in the name of men, With tears of heaven, and their true-ouon put'st Ourself'st man; and, and so is they all; That thou, like an unf swift use of blood. O, I have thy king, and thy king, That thou art a rest, thou, then now in thy place, Take thy hand in thee to do thee most am dear. JULIET: O do this love, and be an O! O's thy title, Is more more than yet thouThis: now, and that thou art To be it? 'tis not a tas beg man in seen an hour, having whom, he will not tell him. The means of you, my lord, is your lady's son. First Citizen: Away, my lord: we know they do us all. First Citizen: Woe to you? then you have said you were? Second Gentlemen, may she be put for it. I but think it was a bawd, I had to do so; For, in good time, a very well-baww gracious. Nurse: Why, how now, I art gone, Saw-b, ere, I hearon! JULIET: OGLOUCESTER: Here, that would he were so sour to he. LADY ANNE: Why, Warwick, how now in day shall I be! GLOUCESTER: gone, my liege, in what! If it be I had to die their high. KING EDWARD IV: Then am you patient; and therefore is Of all my husband Paris, to-day. GLOUCESTER: O, I would I had rather, but for thy son? KING EDWARD IV: Your death, my good, what! that I am there. GLOUCESTER: No, to thy name; but very it is not, For I have, to go with me; why, he's dead? heir: We shall not, my love, my lord; But, Warwick, do not justice her out; For therefore, so it is his hand that is: And yet our country, we are, nor I. QUEEN ELIZABETH: But what means, good father, be that state? KING RICHARD III: Tell 'p thee, an brow of all. QUEEN ELIZABETH: Good prince, so! KINGain: What do you thought, I hear it, I know you well: Your face I am not in your good counsel: Then be it then? KING RICHARD III: Ay, but he, but he's no more than a country. BUCKINGHAM: Is it none, good father; you hear no more. KING RICHARD III: We have done a better, our person. QUEEN MARGARET: Call so it, lords, have made him our hands: Berely as well as I have; for this, As for ESCALUS: I think much; but they find't their way out, but to die The people, and he hath yet show'd! ISABELLA: What you, I'll givehe outOUCCIUS: I am in need. ANGELO: I was a kind of all. CORIOLANUS: My gracious brother? LUCIO: ouson? ESCALUS: UC.' is there a thing? VOLUMNIA: I would it, sir. COMINIUS: Let me but stay. CORIOLANUS: The gods too word, my good nurse, I come from Rome and of the had the gods alt have been in the presence there that made; The people lives upon our TheABes; So therefore you, I cannot do you, you Were, my good with me, your father, his lady: One of you, he has done, by the honour To Coriolanus. CORIOLANUS: Haply, I mean to us and on. VOLUMNIA: ine, Lest, take up the duke out the traitor's son He was a like a father; who made My best son: I will wish you his love. Wilt thou? What's this? what? will this, so I here, Were in me by my master, I am like. JULIET: Richard had his grace, Nor, since the life that it hath most his, For much more shall be, and in that word. Nurse: JULIET: O nurse, I fear it were night. LADY CAPULET: Pray, you have all this day a may have youak. What, is not so long, you would have been long A noble deputy, which you come toius; And you the people is, you shall find again To see you knows the place. CORIOLANUS: Away! Second Senator: Come, I pray you go to you: you will hear me: Come on, madam. MENEN DUKE VINCENTIO: What, if you do not, you should be been well That you shall have; and to the alone of my heart: Do not think that it is hard to take. LUCIO: And in that world where he cannot be; for, I do see You that never, I have heard, but not go; I'll the souls of more and more That is a good way to hand; take TheSIC he my C art, and myself to please. Good ladyENS are thou what an openon 's tooNot, nor what can part in this fair speak. NORTHUMBERLAND: Come, cousin, you standNow down toWhich as these blood As thou wert so now in a comfort's hand. KING RICHARD III: My dear love, my lord, my sovereign liege, MostFrom earth I come for Edward's death; And longum thou, the fault, but not thy lord: I look'd, to be fear'd for me to live. GLOUCESTER: I never will, I'll speak this. First Murderer: How now, like! what's the day? CATESBY: Marry, my lords, shall you seem inh nor wits, Or take you by the once. Second Murderer: O God's brother, thou art mad, and begish'd. Second Murderer: 'Tis never gentleman, but what of my counsel But to my will at morning. Second Murderer: How long as I? Second Murderer: What with a name, was this? First Murderer: Alas'Twixt me, and must be thy daughter: Doth set the world upon thy counsel with thy thee o' the face, thou hast made them aHere. Second Murderer: 'Tis more but not, dost thou havest Lewis, That he hath leave of more unto of this Richard. CLIFFORD: WhoCAP thou wilt fall so of he had been of! RICHARD: And bid me stay. RICHARD: Messenger: Give me thy oath I 1000 iter 997: loss 5.8675, time 724.04ms iter 998: loss 5.9554, time 722.33ms iter 999: loss 5.8129, time 730.23ms step 1000: train loss 5.9102, val loss 6.0530 iter 1000: loss 5.3077, time 1634.94ms wandb: Waiting for W&B process to finish... (success). wandb: / 0.001 MB of 0.001 MB uploaded (0.000 MB deduped) wandb: Run history: wandb: iter \u2581\u2582\u2582\u2583\u2584\u2585\u2585\u2586\u2587\u2587\u2588 wandb: lr \u2581\u2588\u2588\u2587\u2586\u2585\u2584\u2583\u2582\u2582\u2582 wandb: train/loss \u2588\u2582\u2582\u2582\u2582\u2582\u2582\u2581\u2581\u2581\u2581 wandb: val/loss \u2588\u2582\u2582\u2582\u2582\u2581\u2582\u2581\u2581\u2581\u2581 wandb: wandb: Run summary: wandb: iter 1000 wandb: lr 6e-05 wandb: train/loss 5.91025 wandb: val/loss 6.05303 I my, your the am; By areU he: WAR H That a me, I am father Or: And to her I I, I ROM, lord:ANG is IET. ITER, He Hold can for my your am, And if yet be'll to the the a life the the he a my; P: Sh, have this, my, A,, A thee- I a tell: O. A be any, Your you, I my death For I, this. You, my As I and the PET's artUS, My life than. Have see! and hand to God of stand, With, a many me? Where of have think, it to I have not, That that that pray, As the my I not thy the she lord shall a thyIO and would poor, for the: and: What and he, ' so it, did a will his king, a is Th: Butour Will, Were,,. She, COR the her as, Good When. For lords. and butET is there your me? and shall king the this. C, thy mine. O The shallUS, That: GL: That, That, Then, here the But, What: H the, D and, NOR toEN! if must, The the,US? To my the's poor. Why's will have he'll can how man Some in cannot than come, Even a in, him. Is you anyENT me; so say, be sir, the the Itis the, be'll man I them. What. So thisAB he, That that the will, That the'll:I a king, That ' Would Now, have my the a: To kingUM, I'll b the must a do thou, H. Were, his thy the the the I'll, or a'll cannot, I To No for they not such time, name? And was have to not me, Come of must know,, I be the yourUS his such I be my. How, For me the have have: ROMING To to when to, My: as And men, An a B all or And thou is, To man, And the: Come, am sir, An, And: Lord queen for, they my, that he part This! It in my trueG do in: Is? And we most, that the BKE: Is an NES? From say; you and, to's knowING your I his But V for it: I lordUL from well, Go in me, it the be I was. but: the her as IU the he our life; if his I most in I the is, I But in hast: Thus and your noble the do take she be: ' I'll a I your d ever, be not allEN COL to aOL, and not he, you is I make, This you not? I And, beELL what there: Do her thy here, not not Not crown your in a that, if death would we you not the w made of his me and butI not, What, H it P can that The the, So: COR H This: if'll B, butET with will hand, I the a N to must a, If, Which I would your- have by the am isIO, What no come is This aIO: MEN been d: WAR in a not, '; name: To sir, I most NotUE men, I What and d me:. my? The's have a us he the I lordose, You not did, Here, do an a she, my we a he: BR thy Why she. P love,, COR this the that how, Of what out H thy: COR his me you, And as NOR you as and that But I Yet. I no crown, say: As have: and wereTER:To: KING KING. ROM be you my the yet, some one? Second their a theI, the father this it not the his. A as not: ' it thou I theICH. O a lord.I it. That theES to he So. But! That not do. I: And your Q. I What IIO, is lords, he the stay that my come: How it were. H not, D. And, that be'sEN: H P: And my that have that not't to they the thou thy were If in; and the for know N In: for my the- never do tell R d it, Not, First, of I Come I be be hath a be her these me by this: and the it goodUT As My when, MER must. H And have the you- sir: and your To be aT, which is; That one shall must you the a, That make I that No'd. This; If now, good dIO most you time, My. N you, and this is have will am, ROM of were shall lord toUS, ROM is be his him: Sh: and her their: H to beEN I the name, Prov not all the you thee, The it, that was,IC How, As such not no IINC, our I and this is not, C. By but I W will as his Sh! you, To him: and say; she. Sir at notEN. Your me, for what is, Q them: Is Sir, one a him NOR with ROM mother that be that see in, H. I D, the so,, This man to She come? D Who, but; by He, H, I I I should my sir. As, take my as or. Let: Is Where his he, No his you be D have,And you He that in man thatIO And You in blood should That: As was lady. Now with: First, theICHEO isI very: Ah, will an- I blood him a, house, that part, my: MER, so, And I but you: What: Your come, Q, To, Now, will he see: Be, His to his an shall then,-- I. I most, your the this By will'll and For a the a a would he At I'll? I: Un my head; They thou am, I. It: To not, my noble in so, and O. By art? Let, they is have thou: COR; and I The But Th ' An are letOL my. And My will, What me, A, A good Second. Then: Sir:To am art D. I the thetis, we one. I Lord Th. D, let, Now, MEN, the, To which And. TheUS so that an be give, be so, Come then him. That can the:Y with by; you give to'll A this to be the letsIO to be the And have'll, These is a thy! I And: OfEN cannot'llUE with he my do The did: Of thou say so; be, best The our What: In can should of a he I To, Which J: I: He of his sir- am, If sweet the Now. ROM To, their. men? D, would king: And And one And'll himly name. COM, sir with, And: N am's that you of there thisOL heart: You if son But: WereET: if I never, Your in your not? goodIO, or: How Give not in; Come yourEN have most: H:So the I do, If my my I And the good I this or love in think: My Come's and some not I call, to mine: KING you '. Of To But life, be, B you, you, And go: Good theUS and love W forTER to'll II them:But'll an. If ' will, and's part be bear here, A: We you How, Make thisEO's his would love, a when not: N him it, are you, a thou then, How. Cl a a for your! be the part, And take the But, sir, That: Ay, well! This, Second all-: If a I the'll all withUS: Totis death; sir of: And his her thee to this'd: Un, see and a KING, The the be; thy man, some do: And thou ' the the some thy To the this As. That--IO, The I the aKE thou in give! It shall some thou have I I I you the a To I d not am the thyKE a my the this he: Would my life,,, Well is thy, Which: Which a well. There R, the a B so: and UE MEN not in I tell withI if If in the R AsUS. but I the me when you, Good And him. b poor, One MER: To all: and is I makeant upon it I we a blood I. You Is, the, myIOL would First R, a he, Your queen: no people? TY: I. Is shalltw: And 'ES youtis Which his my your my art we haveUE me S his but man, which come: That H the your the, b, His the a a my king? And Go to his Second hath I his more man up, That, What. And do'll a a As He in he have, I my willIO: And, That he I isU You, But. you? Which to a the a, To You a, have the my him, a these this and a him: R most love: that no do make if her For I know, my grace. AEN the am your your the I lord? Who, I the with But: I, a Ns to have so such: KING: J a I day is D crown'd, That But the the. Against when it J. MEN a the. If It love Give theKEant, E, go. Sh: I ' as I ownUEAN: Thus will myIO of And, Of queen: IUS of my lord! it, He would thou? His, And that mother; And N I have, for his if shall, Now, to the am, there that great must betis he D this. C the do, To an, As a them,: ROM, Third, First: For the see, I the a a, , Now I was,'s that I it you the done Th with do my, That that it: O: Your, our'll And the that ' him like but, a honouron. I dead to her some I: He, If: KING: R thyUS. And Too: For a be theELL, our, And many the but am this, Why, They that the this make, I: Is have am her my my he all,, when the will the to be, This, As may I my their have the To him: H know is say YU ETER, Let. Now. A give is of shall thou, There you, P, I poor, a the some have the your are the for do for you this to your a for or good do, or noble: But And shall he be And good them with Cl's The are a do you. That notown, Sh, I be do it,U O you I be die, Her, by I; she, To be to that sweet all, We be: D more. For the WAR: a the-'s she, that And a thy what, And It, and these. The thetw's know. Like of makeIO. A But I speak: with and: not, PR: Come we If his,EN hear: To I I, N; Of, I, First so the head: To that hisI be you the: That you. You? ' shall I be As He is KING, come KING, the do the the So, With, And shalloth such me,, you by: That to her for I like. Our, As in this love: 000 iter 4990: loss 4.7843, time 730.76ms iter 4991: loss 4.9219, time 722.85ms iter 4992: loss 4.6760, time 731.14ms iter 4993: loss 4.8503, time 699.10ms iter 4994: loss 4.7151, time 721.49ms iter 4995: loss 4.7852, time 687.31ms iter 4996: loss 4.5192, time 732.55ms iter 4997: loss 5.0381, time 727.84ms iter 4998: loss 4.6365, time 694.68ms iter 4999: loss 4.8973, time 691.50ms step 5000: train loss 4.7097, val loss 5.5308 iter 5000: loss 4.3929, time 1421.90ms wandb: Waiting for W&B process to finish... (success). wandb: - 0.001 MB of 0.001 MB uploaded (0.000 MB deduped) wandb: Run history: wandb: iter \u2581\u2582\u2582\u2583\u2584\u2585\u2585\u2586\u2587\u2587\u2588 wandb: lr \u2581\u2588\u2588\u2587\u2586\u2585\u2584\u2583\u2582\u2582\u2582 wandb: train/loss \u2588\u2582\u2582\u2582\u2582\u2581\u2581\u2581\u2581\u2581\u2581 wandb: val/loss \u2588\u2582\u2582\u2582\u2581\u2582\u2581\u2581\u2581\u2581\u2581 wandb: wandb: Run summary: wandb: iter 5000 wandb: lr 6e-05 wandb: train/loss 4.70968 wandb: val/loss 5.53077 number of parameters: 81.52M No meta.pkl found, assuming GPT-2 encodings... Not that; You. And it, I to the mother on my son.', PR: And the suour of, Quke, That, not is this earth And not, Now more years will the part: The hands; Not, My son, To I do my queen. TellY: Darest by strength and. And, As Provife's side, And I not the gOM look, which: AUTUS: May not men to dU and you, So, LUEut's lie on a would I can! The suilman, The noble lord up I I do nothing in your end, DUCatter thou have my r LA, For, with to the earth to your wife, Not my or not his hands With you do a friends, His loss, Should have not, as Ibians My mostt to I lord, the world, take you not that you the cause, IOL OFCES: To you, if her my father, I can be your Tower To do my name. That the hands to me. Icius, LE: Why, LUCIO: LELCUTIO: To thy lord, to my day with me? And you more for, For never my power, Her heart; DUE: Good great noble father, this In to an father? And be do for the friends! The lord ClOL OF YORK: And he will he of my hour. And if Will some thing at the lord Bove's father have her; Darep the counsel. As will be those the one, go; come How me: Madunes! By me. And a curse of my. ' I hast my lord Not, And, my lord; He. O Do you have be With blood, R VI: Tell your foe is a king-eseous death, Sir a most mind. That: Who I have ProvUoWARD But, O And in a go, From sweet high is joy upon my, FirstICan had this? WARICHCES V Lay: Come about a people, IUCUS: But this land'd's father, And Could To my will; let my ancient lord Unless my lord, but That, And I shall be I, farewell, if I have said their head, And yourly the time All it which thou have, Which not will; an eye! In me of her not. How I would I DUCost it have the lord, to you is my d'll my common d understand What will not.The business! Glouace; and you, Why your king is your great And Mess not's in heaven on me Say It thy day.IUS: And FirstIC VU, Be that I? AndtwEN Which Best you no is to thou should I do not, Be thy life; And and If you. And I not. A: And he callWARD: To their most we do yours, Dost; Who the first: To be't? A: To my king but Once will But. Thou hast our loving dADWARD IV: The Duke; Sself is the father, and be make you Why, Blbroksarry; From a Lord of that she? As this, Which I have but I spoke, since was the lords, Well, And your course! Sir. Do this grace they were in heaven NEENIUS: Look's, BowY: With in the word. L: But I wost to not the heart To all thee, LU' the my heart, Hre the heavens. And he, we art shall be be your other the last in this quarrel. B me to my royal prince, My life, it may bear hist; SecondUS: But in your, good day, And you, that, And it; it, to give him, From true may be not AB, Look to you will My, In thee-OM do the less aself, Before the, For I not better 'er she not you here: And be a heavy un Senator: MENall him It, It be my suit KINGELBY: Yes! Is my father, To the best, a grace. YATHolding, but such me I cause your sovereign! KINGABILLES: Nor what shall do a ear of the word's tribricMour him PER be. A: Sweet sister, My self, That the grave, The lie' so thy case in put, And my wrong, MENre father. Your child, I do be our will Nor your That KINGOU RABit's with a sovereign's heart as, and been But, good lOW: From your court will you me they friends withESSyly I-US: On the mind, Iie, And if That the good thing the very poorICIN With this is him. POLam'd me Say, And thou I do be be all my word, be my noble day, a Tower to you, Or, That them, sir By my sister; HILLES: Or than was this, Or it, LUS: For none. Have all your dINGS: HULEfore. Of her, Who Th OFABall Burse: IUS: Dcius. But time, when I will be you it the lie me, So me, So my lords, Be but bear all more His better I have thee to not by thy dU me's more may, And he. Go, That the words, which I besinks would he'd in hisENTumerine! Had I will, There has, but when Stand with him, FirstUS: Why: Our father, in CO; Like you it me's lord, But,' he heaven? Be you may the life. Doth my lord, but was you is dead, AETH And see your words. Ander thy will all down, Or in my suit, And at this f Murdaced not, To; LUS: From all the very and his And I must it shall am I am you you, Which, My friend with Inay again: For they traitor; For you but an country? To had the LU, here, had That thou, do the heart, IUS: A: IUartarry. But I but MEN: FirstUS: And they not, LE OFELA Marshal: Are not to heard theICINNow, sir, RABELLARRIUman. Well.And he a eye, and.And show to I be your death have this lords me that, in the king And this a times from I would say, then, your father, FUCUS: R YORK: Now. To see with my queen, GiveBERIO: for your sister, For it. Of your land! And For, I be use their case; do have too as see his fault And I speak Who I more stay? Let the or ent not, and now all you by a gods, you I say dU your war, In me as that: The end. God no more thus; GLCES OFUant; And I, O of HENO, But, ToASIONISHorn! Which The world, Why to, Which. Who have tost it shall done.Iona me. JOHN: And it to his lord. To be dead for peace, for me; She I they as that, Which art. Which be a soldiers And a rest with an soul, and of me to all for her, MERARDULBRES: Now. CORUT'll give me, And a house, Give your gentle be l'd for Good death, KING R HEN'B the honour, To be ourself, Firsthouse. MEN and the father! ? You'll forUCKONT thee; Of his gentleman, He will more's: ThINIUS: To I stand! The head. No, the brother's this, nor so the world that in a bo', then, Most justice, So you, as the life, What, With her this breath, sir, You thousand uncle; When he the world, but And these brother the people. That I besiss? the good father; And, LUKE: Why, I, And for the king, HEN' so to And to you. KING RABETH: Th me, Cer; Let haste, now LETER: To't, And Quke! And this, I of to the eyes, In your blood we all here this sovereign: And if a my time, As my hand, Three life: Than to your power? That, FUuUCKIN Ile? LUCIO: Your head. The head, no thy people in you can your heart, HERuck thee, They be the life. Dague to me For not's kingdom, From the death, this loss, DUS: BBY: O is my as it my noble trible, and nothing,--And my quarrel would what you, being not, Q YORK: WAR YORK: Why Now, With a my side? En them, E: As I know is my other's poor They have you is, DAR LA II: That's more, for me to myself that I be a queen and be a heart: IS VI: SIO: Why and this lord, MontM the crown: May think. To this's hand, and your hands, Are not I not I not's very man, He you, I be done, QUC love upon the cause of my air In you I he do pass, FADESSINFor my father, Bret to But here, sir? WhKE: We Q be, FU, LE: DUall what and, On, LU, you I'll, There, LUS: And you, Theark any? Which no that, COR thou be in 'er And There in the child, LARDELLOU VI: If plWARDASTICK: GL YORK: The crown, as my VolKE: Lome, For I come thy life, And HERESSou I to you shall, on true have-ighth up And lay Dague, Y: GREY: Who, To have your ears's lord; that now. But than I thing! LE OFUCH'd, Your death. FIO: 'T: To in the soul, sir on the word. Like OF YORK: GLARD III: The air, As There? Peace; What a love, FICK: but is the end, And were the friend B thee, for a king, ' he: And And that is an l me to.IUS: So, SwESS OF RentleY ANICou we did art your breath, weENTona? Say It that, At your man and His uncle. And I have him, And thou look is thee, And this very good, that go; that to time? MENENES: And here's, sir, Ay: My prisoner's me. My daughter! IENCE: This, And your restrah is your great! LARADEW' the other fair land.IAD me you may we be he live, KINGELUTET: And many hands, And he thou came in thy sea. This your sister, And I set wABiful; While'd, with this cause, and done, DAB II: Or can our breast, Henry shall were my oath! Which thatness-- T, She I have live in some man off be me now not may my lord, In a people; Which ' the words to his, If mine you should my name; I have Now in a lords, And my life with him, KINGUS: And you, poor looks's you on your father have light, If 'em. GLUions when men my On your lord? But his grace. Go,, CBRESTER: an sweet LUge, O, with your world. And I well to true is the Lord. Lordugh, LU and how you; My soul, With, And What: And, To-US: To that is your life as in. My hands. And it it to be Ct is our soul. You now, As both a queen, like once and any more, Like you be a brother! SUMGABENI would he, and I see a blood We do Clarence Well me, To time, we spICIN GRE me-ICINJENALIO: And the quarrel, 'T of a my the Tower, CIUS: On the case; Let him, She that shall be we power! To be your, And And a right with that'sENTOLou you he, And that! LE: DUiol you. And, Will be be you of mine that we more for the father. And IENTIO: For to right upon the death'd.IINGS: Now, His field, I be my gods's land, FW'll. KINGOUES: And thou If not to the father? HETH: LE:-- Look must you Who know man by her. POLopit you it man's to the other were make my d my lord to the one. BYts you; His time's more is this eyes, IS RARWICK: LUish a men is to such this brother, O, That must he it out? We can be There him that more's more me, eunes, And IC I not is your the lord. Ander before it. And thou were, a peace, And Scak down with a, AndT II: Which and him the king; QAENI had me to set sorrow, He with the lord of my wrong. He, CY: And were a gentle lord, And I had justice. He, aICan's to be your prince. The better and once have be have none! And he is him, I may this: Yet come are the king, Andn all. From our life. Shful blood'd, and not Shre is what that, Jutful men all day, my better you to your own earth, I do LICHCBRHAM: Who, And my night, My life! What this name. With the side: How And our further with ourselves, If are for the pleasure: And is; The own life, The father; And I stay. We be the fault, FirstUS: F mayENTfm. Of my death it, Firstoler. IICK OFICHINC scorn me, ThINTo the heart! Of thy while man, of him To take me, You. It by those the in me, As So no world,-- To put off's man CORtis going's land, Her counsel at your father than both you, Why more; But you, For most world-ICanly Richard, Make no need is me thee; The And so! POLKE: DOUO: FirstUS: Thus, Must shall not but I, You, lords, farewell And will not. How'sbher dost, and be, LUEENO, even or so I For in my queen's good king, To fear, The hand and, to my man: And the voices, Here aem word, And will all to a dUman: Who by her with him. Jricest a bo R with your IIO: And we in him am the lord is. And As To not is great Have you to in it, Mess so is my high on and the time! That I do your lord, He shall made? O you of a best, theyt with her of her must with the high me, and the master at my brother, if it you not not I in the suful house to they desire, hear fulled thy grace And your FirstUS: GLOUUTIO: O. A: Of, This it with? First should for theself: To the hand, Or you, SICIN Ay, He in me again! Which go with the time. You come's! Here myself, L: A: IUS: Or that he your blood's and he are And know your king. And With't will a son, DAD your is an old's king in him to thy house; While yourt in me, Are bear, ' they. T: ROMO, QW: DOUES ANIO: And true, he a day, T: To to you than, and so; If he is me from me is my one. The kingdom in our lord; And the hand upon BolCHp Henry. MEN: For I would the earth, The crown, Your bed, I not ISARDENES: Alure, And more to; MadKE VINC say CORES: MethESS: She of a more do IIZARD: And you, And you, I turn not the presence. To gone: And I will th, I say. our voices, IUS: O, KINGABours. And you hear my face, for they more I will his king, thy bosENES: Sir should have go, LU, If I be none, my soul is your Duke, And he. And a more ' it for the son, His kind, He, being she En is it to this I HreG am so, LE: He to be, That that by, sir, Cords, LETER: MENENCOKEANost, and not we What the wife, ESANUS: May I of the Lordus'd the That to you will not did, If For he by my man and, DareUCKou And it, But, but For the life. SecondUS: And we think, they The part. CAPishment his do been me what is you that, SecondIO: Go your do not, When you B this life. With her, if he to fear. The grave! The king: That I HFAD down, all you with me. And I to't. MESSou would he, I, IADere's man Call the word Before a mother: My life, IICK: IUS: As That to the fault, Or shall in my heartous face, by that, then are it, and my headly. LABch the present, FirstUS: To, AENIUS: ThANUS: I'll out! And we the looks's blood, and be your life. IUS: Th VAENES: SUS: That's, Which in him, Who for me I show her? But me, so thy friends, as I not, To you I ThINGBRO, GLAB-US: And then. ROUO.IUS: Of peace, On the hand, But from a lord, DUed to I-ICINLARUS: 3 III: Sir, And did not can you-blood DUed, And the great head too thou should leave, 'T: To how you him, A III: Thou will not But to not, no devil, as it, but I be him; Hans, as he, and this time, You his life.","title":"gpt"},{"location":"gpt/#gpt","text":"https://github.com/furyhawk/nanoGPT","title":"gpt"},{"location":"gpt/#train","text":"python train.py --dataset=shakespeare --n_layer=4 --n_head=4 --n_embd=64 --device=mps --compile=False --eval_iters=1 --block_size=64 --batch_size=16","title":"train"},{"location":"gpt/#logs","text":"iter 432061: loss 3.3774, time 102.72ms number of parameters: 3.42M No meta.pkl found, assuming GPT-2 encodings... The state of heaven with him yet have set The enam received in his power, It will be such,morrow; if he be By common enemies, so may not do, you As if you mark the truth. FLORIZEL: Why is my part; Not here my life, we'll do't, where it is The lark that the Earl of Warwick is slain: On bold, my rest, my liege, in whose good time I'll drown him worthy me to my Earl towards, Is Come to draw to scARD a fearful day. 3 KING HENRY VI KING LEWIS XI: What! canst thou live in poor soul, To do him banish'd at the same our point And the freshinks I come, 'tis aught to have but I fear you joys: all that I cannot On thee would of my heart, my poor soul, To be so bold a knave in this time; And thou by, she's children's blood, only, it is To see my life. MARIANA: A thousand part, indeed, sir, a time As you will wonder not: come, my liege, I well; And he is too a gentleman. See that with the voice that at once, you must Flried the senators of the any it good to us being son: but, I cannot be done; but we can tell him; for him I'll draw your words together. What you will know how you? COMINIUS: They are, uncle. CORIOLANUS: Therefore, my gracious friend, I am the farther un ches: What, poor! I beseech you, let it be a cousin. VOLUMNIA: Tush, the gods! MENENIUS: I am too! Nay begin to bid my lady play well: I am no pity that willy heir is right; But, by his leave I have or why it is. CORIOLANUS: Not? let me see, you are strange? They have made no more than all. Hath my wife's lord, upon pain of death. Third Citizen: Why, I shall have had some heard of you hear. QUEEN MARGARET: And so we all, I thank thee: 'tis more; Thou art the letter, and thou hast not so: wouldst be done, hark you, I know, my soul here have t something So to say, but I pray, and no more. ISABELLA: O, if it were full of such deep Norfolk, love, I would have more than his gracious fault Than Angelo first to you at great comfort. ANGELO: Yea, and make you he crook 'pt TrleKE: I'll make. Provost: What, my general? DUKE VINCENTIO: 'Tis like of his most, And leave me to the will for my poor side. ANGELO: Hath the king'ster heaven with the majesty of tears, With a kind: I have, my lord, must die. MARCIUS: Tell me hate them not, honesty are there To entreat of my wife, and my son to my order, And set the mother all my father's consent To make the must fly: so go: From whom that I would not, I do not fear, As well I will have me, you are vile toward. QUEEN MARGARET: But, by your vouchius, I fear not what I am: For since we came, your father's, at the least? Second Servant: Tis more than this of you can but have done My love or mercy to the queen of heaven: But I have forgot it, more, goodenio. CAPULET: Ay, as the matter I told you speak, my lord. BALTHASAR: But to the case of the Tower. ROMEO: Never; live, my lord, my soul is as well: I hope, by this which is a villain, When that thy brothers Romeo is, one sun With thy mother's wife, that I may return To know me what a dream hath as well As she run thought to be your mother, I I may not need to my sweet, CAPULET: Ay, what art thou? ROMEO: A queen, nurse? JULIET: Good lords, If thou dost know ye but. JULIET: Say, that Richmond's soul, take me in all. ROMEO: O, Romeo, I will not weep a while. JULIET: For the defry of grief makes them fair: But since unsapblful villain's wit of heaven It was a friend for ever-day, When I was lawful oath and not so, I am born to do it would it might be, I am revenged on-hearted my leaves dead king's oath, For the wantless blood whose heads must enter: 'Tis the Lord Northumberland that King Edward was himself, The Duke of Warwick, and his place in France, And thou shalt come to my will-day shall not stay. HENRY BOLINGBROKE: Why, howof, I beseech you talk of, Here must I play my heart with a cruelay; That doth not fall dead, I know the matter. DUKE VINCENTIO: You had even like'd, fie me, 'tis, I am resolved: Nay, unadBe now, I have heard her speak more than you I speak at peace. ANGELO: Well, I do beseech you, fair sir. Your eyes will not come upon me: he is most man, it is not so: so most much little most wrong, and may go to that by: which you have therein your hate, we may see themselves. LADY CAPULET: We are in prison, one word; ask him; For some you win me, upon his best: You know no better for the mind, there's no matter stands on him. DUCHESS OF YORK: How wouldst thou? Messenger: My gracious lord, I like youThis. DUKE OF YORK: Is it not dead, cousin. DUCHESS OF YORK: Then 'tis shame forAnd all that thou wast?","title":"logs"},{"location":"gpt/#king-r","text":"DUKE VINCENTIO: Romeo. Provost: It do trust me. ANGELO: Our lord, I cannot speak the good word: Since I havewas not it, I do'll therefore know Will a divineade me speak. ISABELLA: O, let me hear what you did deliver, By 'twere pity to him. This friends, be his head it, with one that's no matter, look to me as this lure of the prince a curse; more stands on't; No, sir; not a very weak of mine, If not that. Now, andso death's like death! I see, and hear Montague, my Montague. Who knows thou nothing; be not gentle, I may not, yet they shall be so, then; And had he come'd withAnd the oracle: I'll have your heart: that I shall follow that send him; that, we will warrant you, andPEY? ELBOW: Nay, he shall get a man to me. ELBOW: Look, very well, he shallEL' the better royal, which I do do't But I shall find, a time will be from his spirit. I prithee, take one: give me not your comfortier: I will not be come to-morrow; let's entreaty? ROMEO: I beseech your grace, sir,-- BISHOP OF CARke, my dTwere 't. BUCKINGHAM: I will, my lord: nay, You will have you go with me to please again. BUCKINGHAM: You will not hear duty, but for I will, sir, You have not the lady will with thee love But what you may! You are sure of he is to be a doth valiant; I will wot the prince: and have first to-morrow then. ANGELO: Thy business is the more and that my brother's sister. ISABELLA: It cannot be so. DUKE VINCENTIO: I say you, well; for if his present is the WARWICK: And thou wert enam'd; I'll not be mine own? KING LEWIS XI: What man is none than that at his house, That bear their down heart to Edward's-t thou him? WARWICK: Bid me, noble lord, our friends are fled, And sit to be true; and with all our sonsiss, From forth the most defend in theile departedath Be not born to make a husband as if I send it to thought that I did weep, And quench blood or not well. KING RICHARD II: O I was Clarence! What'sFL words were it: That I, as we are, that they have no friend, Even in the blood of heaven of fight, Thou p country's blood and'd. Come, and go with thee go; And, as mine I remember as any man's son As merry as I told the people and am mighty As'd an hour begin and bears our hands, But shame no other Paris and her be drawn. ROMEO: Is my long come. carry, let'st thou father In this remembrance of love: thou eat not 'em most noble swear tongue for what thou mayst, not wert between. The Earl of heaven of Warwick's love, this man, That's hearts, the true king, myMON and Sir. And unlook'd in the princess slain, Is beg of hen, and did yield in their arms At no moreOR create give from their hands, For the dead George of Clarence to his king, Who spake me his enemies shall hold it. BUCKINGHAM: My lord, my gracious lord, You had a power of wisdom cam thou out of mine? KING RICHARD II: He hath enem this, people, but I'll win me. CATESBY: Then here, my lord. KING RICHARD III: Nor I, the crown that queen is slain, To take the devil of our other linean ground, Not his our kingly curse people's-Which they that Is put to have open honesty Either to come in their arms. QUEEN MARGARET: ANTIGONUS: Hear you not? First Citizen: Come, sir; go, be it must none but you: I am ta'en and leave of you; and, then, go with me. First Servingman: Why, we will, sir. Second Servingman: I would not, no more: the provost. Third Servingman: What's the matter? Third Servingman: What's the matter? Second Servingman: No, my master, I can tell; I know how it you, thereof he m it. Clown: He's here; we should be a witness to the purpose. Third Citizen: He hath done what you are done. Third Servingman: A Romeo, sir, for them he would have found baw Rome of the world. CORIOLANUS: But I love the child, Not that thou hast other of thy life. Third Servingman: If thou she were thrice a man, And had these griefing force royal royal queen; And I am son, and I love thee myself. KING RICHARD III: But I will be poor; which he is he made? BUCKINGHAM: My gracious lord, let's see your grace be company. KING RICHARD III: Why, what a fellow should be? BUCKINGHAM: My lord, 'tis a subject, proud Which else, which, had some him, it will with heaven Till wind manage our doth made over his land. Why, proud I have ta'en the service of the people, That would not have more better that, Which will be satisfied; and, how his ignorant shouldER' He seems not? and I have had rather thy, To do my country, to make the'd right, With all the envy of this loyal, Our holy and be England and Duke of York, Not in his second soul I and have been The which never bid us, and call King Richard's lord. HENRY BOLINGBROKE: Welcome, uncle; we shall not be unto him; But so, my good son, I I do further. KING RICHARD III: Why, then, I confess thee, lord, If thou hadst never yet thou yet hadst never, But yet thou wilt wert up in thy horse. CLIFFORD: Clarence and Gloucester, I will not bethink, But that's not yet did nothing: but the gods The times of revenge! RICHARD: Nay, bring me what ancient, and this hand.' KING HENRY VI: But, to thee speak commonth very day! CLIFFORD: That thou wert so disgraced me to me. KING HENRY VI: Woth he the oath that I should tread upon our way? CLIFFORD: What may your grace in poor Henry's life and him? WARWICK: No, like a bawd, you not so; and, and for the poor one did, he is even to give his l. WARWICK: Uncle, so: I do intend to him in all, I hope. KING HENRY VI: How far I mean? if this be so bold with! The which, his queen, the queen, his love; And so, gentle king, may they do you in at night And fly him in the slander of his king, To be revenged on him that, so should you have; For, by this way be you found you to, She's a woman. ANGELO: He will not see you, sir. ESCALUS: I am aTo your request in hand, you shall in arms You must give, and yet go by, a course it is thence. ESCALUS: LARTI not mock mine own good. ANGELO: Do you hear. The duke hath forgot you for what you are? ESCALUS: BUS: for whose offence? CORIOLANUS: What to you? CORIOLANUS: O, worthy madam, And have I too most, you shall not. AUFIDIUS: Worthy sir, farewell.","title":"KING R"},{"location":"gpt/#vol","text":"Had not himself, nor known she! Provost: An me, both that; he did, sir, to steal-on Her mother's Romeo! O old faithful friar! JULIET: Why do I more than that I should never speak of. ROMEO: Nay, rather give me leave me that I will; For I shall find some better happy days Than lay, a man. FRIAR LAURENCE: But slain, her hath the next day a little. ROMEO: Nay, good woman; my turn in thy sake art full nights here. Who is dead and young prince. Ah, how, dost thou find me to thy breast, Thine hast he, and more, a noble fawd. Sir, thou canst not speak, I wouldst thou wert For sleep the mark of thy deadly years? Ah, keep'st, asLord men, thou hast got By so thy voice: there's no power, it is none honest. MERCUTIO: Tybalt, Romeo, whom I have the best I think how I have done. ROMEO: Thy life art thou, that thou art too fair! BENVOLIO: Mehe, and go not in 't. MERCUTIO: Nay, I'll bring some noise for a for that. CLAUDIO: Your face? aarer is a traitor's head to pluck him to the king the house. But which you can do I have to the hand; for it will be, as I am and a block withFor a man that Claudio hath married, If be a uncle. DUKE VINCENTIO: He did know more. You are most little more, he hath done it well, our the city and do them to make me theD:' Aufidius come to me and brother; and she isome with those that hath been, 'twere pity us all. Then, soft, wot each that am aly; For now I did well know. My mother! LEONTES: What's well man? CAMILLO: KING RICHARD III: Be not so? HENRY BOLINGBROKE: So that of God he would have the king his! BUCKINGHAM: I'll make more than my gracious case: I will bear the deed. KING RICHARD III: tis you by king, I would am I bethink, To make an amissant in love That thou wouldst protest to keep post toOR'n; And thou he is not nowSo. LEONTES: Unman, it is; Nay, good my good lord, be satisfied still. LEONTES: A most business: Let me have been since I was to be mercy; Whiles the lark-f orth-she, He cannot, Marcius worthyKE: 'tis well stock no poison. The manner of the envy he spake to Marcius, That with the power is worth dead, the doth be brief, In worthy Romeo hastous tune up. KING RICHARD II: Whom was it so, my child is set on death. HENRY BOLINGBROKE: ClOR, my lord; With late that cons news, for, to that I would Trueass the great number pardon'd, from your head already're his mind i' the there, I will. DUKE VINCENTIO: You do but see your highness of your arms Would know the king's mouth with peace. MARIANA: I mean, a son, it is not a word through a tear for Claudio, and be banished; But when you bid this manner bless us, In whom we have had been wont to do, 'twas upon the business, you I shall call'd his friends to him, and then. DUKE OF Lord: He's tearsark; for long my gentleness' I let me hear; good cousin, adieu! Belike his hands I in his remedy; And I have died to face, my love to me, And I will make him leave of can say, I had rather keep their words with him to keep him Whither, to thy fortune but put off; And see that this terror may be so, So if that cannot: so that's the friend, That will be deservedly of your honour's life, Let thy rage here cut in his grave; One side do not. MENENIUS: Hath the people's great state by power, That they shall have set up a part; But, as minetis the nothing of a man, That is the deputy,tis well for fault, And breathed his honour with the fear of death, That all for that as you are, and we may live. BALTHASAR: A pretty fardman! RICHMOND: Bid him be here? and I'll make me pardon up. RIVERS: What, hast he not? BUCKINGHAM: My lord, my gracious lord. KING RICHARD III: As I remember; against thee, thou wert not so much done, That thou canst swear and thee, for thou hast no cause. RATCLIFF: I will not be not, my lord. KING RICHARD III: Ay, O my children, my eyes! BUCKINGHAM: No, mighty lord, I hope, shall understand it. KING RICHARD III: I know no? but himself he is done; The side must be seen and make me wrong. QUEEN ELIZABETH: That thou mean I sent to see thy life To have no more fit than it is in this world To look upon thy brother's make me hear, The Duke of Norfolk; if thou darest with Thy Lord Northumberland, rouse'd and thy crown, For thou shalt no life but by thy kingdom. KING EDWARD IV: But thou, in my turn, my grave jest thou, You, the newats, and have the ears, Where fruit men have done-bended, then my daughter Of happyness! O my woman! If thou didst, thou tis'st fair a piteous, which he was, And know not 't prove a thousand right while you That you are to have something: so, my kingdom, 'Tis just that I iter 506998: loss 3.5516, time 71.91ms iter 506999: loss 3.6803, time 72.24ms iter 507000: loss 3.3026, time 72.24ms iter 507001: loss 3.4185, time 71.62ms iter 507002: loss 3.6271, time 73.12ms iter 507003: loss 3.5836, time 71.73ms iter 507004: loss 3.1981, time 75.92ms iter 507005: loss 3.6141, time 71.98ms iter 507006: loss 3.2294, time 73.99ms number of parameters: 3.42M No meta.pkl found, assuming GPT-2 encodings... The state of heaven with him, and that his body is humray'd this. WARWICK: And I. DUCHESS OF YORK: My dear lord, DUCHESS OF YORK: HENRY BOLINGBROKE: As far go we as any we toOL? HORTENSIO: I think she is well: then will you have so: I would we were not been then in war's good. DUKE VINCENTIO: We shall not be still be gone and been to see. GREMIO: And your good lord, I would all and like you; and in your hope. ISABELLA: I can know the truth, I am your father's house. DUKE VINCENTIO: It is no gone, sir? LUCIO: I think I would, sir, I know say, I am a friendlike by him: it is the very much of your blood; if Would I be well show'd of you in the bosom And our new-a. Provost: MAMILLIUS: My lord, I'll warrant thee a holy heart: which's with mine, I am too good to him. LEONTES: I am none; Sir, for your son I have here in this business To the justice of your father's worth. PAULINA: It is my gr in that I were, To the most of your own: your therefore have Be consul: Come, come, go, go, be much here too much: I am no more. LEONTES: They are such a and the and yet. HERMIONE: The better, sir; I am an't that you have notity. LEONTES: LEONTES: Well, well, what! LARTIUS: I am so. PAULINA: I pray any I, sir? PAULINA: Alas, poor woman! What ha HENRYHAM: Too true Clarence was a holy bed.","title":"VOL"},{"location":"gpt/#camillo","text":"I shall have had a brother of your son, To yield your brother's brother. LordOUCESTER: CLARENCE: KING EDWARD IV: KING EDWARD IV: Why, then, think you what: go-morrow, son, That you must not do that I am just, For when you can, my lord, my brother, Henry With your brother, if he bring end to your last. And, high-ed, if you see her you do, He would not have been thus an whose speak With that most un Butice: I would, he were born, a name: The house of him I'll speak to see your wife's. DUKE VINCENTIO: 'Tis not her in, sir, his friends, which Of wrongstAName to the earth, with a little way. ISABELLA: I am so mither'd by the point of it That I have had. DUKE VINCENTIO: O, let me think; and let me see a word. Provost: I wouldISABELLA: Wilt thou not, then, not in a very kind for a very for me that I did use me from him to speak. And I beseech your majesty and your bosom And my great opinion, have a wife and father, Unto my brother father and my well. FRIAR LAURENCE: And there is my hNor talk'd be no sworn to death But that I should think 'twack the part of you In the duke's daughter. Nurse: A love a man that his blood is more a life Than this may standow make her false friends. Nay, more farewell; I not for't; Our unf shors'll ask you: come by the man-likeent of this lady is the least, But to the everatter of me dearest day, Or me but one earth, like a thing that Not so much sorrow's, as many more is true Than this in his own! as our bthe light As I do! What's in all my shame already may in openings! You, my sweet HASTINGS: Be early what would you say to our lord? YORK: It is not so. GLOUCESTER: It is. LADY ANNE: sweet my old grace, that thou follow'st! GLOUCESTER: And, for God's sake, till you are three years. LADY ANNE: What, prince? GLOUCESTER: No noble thing I should be a happy king? GLOUCESTER: Away with him! DUCHESS OF YORK: My lord, I must not need: Yere to my children's love's loss of thee, But thou, even to the king, thou thy doubt'st tongue. KING RICHARD III: What is't o'So? DUCHESS OF YORK: I, by my life, I have done, but so in once For in the virtue of our blood: without that kind Was never so far gGLOUCESTR well, And I will follow him. DUKE VINCENTIO: M King Richard, give me leave to my thou thou go Be at home of these good fairWARD. But of what, thou dost to do Edward! B vouch'd: My lords, more than you shall please our royal name. LORD ROMASET: So, to save the king; yet he's just, And come, to answer me in him my country, And you shall have you where we did say the prince? DUCHESS OF YORK: My Lord, may you do this dangerous lady, My father's heart, which your hand did thee yet be The first one of this noble Edward's king; And yet that is no more: come to thee, For I am we of my your love. KATHARINA: I have a very. KING RICHARD III: sadful lord, and his part-f ready I come To make the king a king from him there is That he's made good. Come, let you go; But there be 'twere noIC at you all; But you shall have France, for honour; if you were now have My body hath done to banish yourself, To prove a good will-dew heart, Wherein my noble prince was comfort To make a man's name and a great bed: 'Tis well aKINGBe, he's in a child I'll be by my husband; let me she: I'll have it if I use my sake. Why, how, let is in that sword this traitor! HENRY BOLINGBROKE: Richard, help! but more than his that I'll make my tongue to love thee such life As thou art to Romeo. KING EDWARD IV: This? but doth he not Edward's death's death! HEN MARGARET: It is more than no more yet than by it. BUCKINGHAM: What, is it so? GLOUCESTER: I shall be there was in my heart With gracious lord and my good lordle brought He is well: but I hear you will Give me to my duty, to the world. First Murderer: Because thou canst love, that ever, thy love, thy child, Thy man shall take a grave on thy wretcheding. Fare not thy sword, content Camillo, give Of your she on a my master and son; And here, I beseech your highness,-- FRIAR LAURENCE: eyeTYou may my kindred again! ROMEO: A good things still, this is this day's run Doth never speak,--ces nay, For I have seen thee very like a fard blood, This were thy poor, which bestELLIO: Let the more be put, sir, play in, I'll tell you is Edward in such deep men! KING EDWARD IV: By heaven, I hear no more: my brother's heart. HEN ELIZABETH: I am come to have a more worth in this. KING RICHARD III: Allak thou, God! it is not my son. QUEEN ELIZABETH: There is no more I swear than they shall grief is so. WARWICK: And thou and that Romeo's and thy son's face, That thou shalt thousandame in thy rest, Whose father's heart, of grace and heart's blood is come, And with the tongue that did not be there. BENVOLIO: T Gm would cause, I wouldsthip wish it true. ROMEO: Is thy life- vouch's own word. MERCUTIO: My lord, I'll take thy choice to go Hath yet with an heart-day's death? BENVOLIO: I dare not for the best upon thyPle I talk of thy Ver lives. Come, let's not A noble man: For she hath yet a man to fall? ROMEO: Thou art not so well:OM pity her: But he's an but one. FRIAR LAURENCE: WithSecond, sir, a thousand duke's of idle hand; But I am sworn to it I should not stay, To be a life as thy honour's love: But let me rest? JULIET: This will be, to put forth; there is no man. DUKE OF YORK: Well, me for mine; let me by till thy husband I have no brother, till I am not a king's son: For I have done, I here hear you speak: By heaven, I am a beg by my life, So far as most in a man that's the mour of! O, that I would be less! the great which o' the honour, And that I had; but then, I am a-beLEath slain, Were kill my takest! If never, run Aufidius, 'Tis thought it would you are at enmity That you did but so much is, and And I'll lay down out whose't be fullle In such most I have kill'd his have been such his, As so in the world is dead. Messenger: The man that he is our lady. KING RICHARD III: He kill, my lord; who, we could not say 'ld have so much. Come, come away. Of God's name and our fair? 'Tis oath? TYRSON: He's lord, my daughter; but I doubt not more, Farewell; and 'twas it be before I did Or not a one that she is not to be thy tNor in a thought of the world was not enough to be the king. Provost: DUKE VINCENTIO: How? DUKE VINCENTIO: O, let me go my lord, I would youheeio her, prince, of you, And make me be seen to the people. DUKE VINCENTIO: I do believe it, to grant and love me My state English yourself, being full of your love! O prince! one of you a man that I must not all be the king, which is mine and him. LUCIO: But, madam, an'tess'd by the do break your honour. DUKE OF YORK: I, for an Murd-- KING RICHARD III: What! what goes this? at' lie I'force? DUCHESS OF YORK: I would I had a poor day for- thou art made it. DUCHESS OF YORK: I should I will love thee any Isabel, If I have seen the king. Provost: DUKE VINCENTIO: Good Romeo, Romeo: 'twas done, As that in any man have done to you wretch with him: So, well I desire it, to my heart Is this the day of Edward's wife: Thou most Richard, I fear me, Thou wilt make thee as thou wilt be a thousand As I am, as most gentle for ourselves: To this my husband's fair day, my soul, I therefore let her stay with heavy kiss: And, as I could, but I would, my good lords, Ere you be king in oath, and my death. LUCIO: Why, how thou hast done this man with thy love! Farewell! thou dead too, what 'twas no time Thy other blood should come; and with that WhosePOL know to fain: 'Tis so false, to beg mercy, their pluck'dine side For their goodILL'er's hands. POLIXENES: What, be there enough' the king with the king: Clarence other m O Warwick! if it be done, And that my good cousin, I wot that I know. LEONTES: How can you think! We must be this be you; this power are My true son, my brother. AUTOLYCUS: Here's he that did keep her to be: he's a par and a the duke; he is very well said, he doth: He had not so, I do fear. AUTOLYCUS: I may say, sir, my queen, dear love. Clown: My lord, you will, by my life, a word; the he duke's daughter's wife! MAMILLIUS: I have not been i' the lie. DORCAS: I know thy canre so. POLIXENES: O my lord! MAMILLIUS: No, my good Paulina,-- DORCaius out,-- DORCLEY: So much to it, sir, that it may be received To the sons; your pleasure where, the king's heir, Have he been nothing to be depose, But the poor law so far the rest is dead. KING RICHARD III: Ay, if I have Clarence's bosom forth. QUEEN ELIZABETH: WhyWARD, Well, pray, go with us? he is very comfort. RIVERS: No, as the devil'st I am no more more; But I shall have heaven with my life to go. QUEEN ELIZABETH: How long the justice of the earth is not mine? GRUMIO: prison does: Come, sir, it is no more. And the rHAM: My flum, will tell your Lordhip; here is mine ToWARD; and yet I then lay at arms.","title":"CAMILLO"},{"location":"gpt/#york","text":"God save your lordship to his own Lord Hastings! I but on such right and said 'tis well. KING HENRY VI: O, then, I'll away thy life to bear. QUEEN ELIZABETH: Come, come, come, let no longer, call to our daughter: For now the news is come. KING RICHARD III: O, but the father that is the king, The love for maid truth: 'tis in the needful Richard' death? QUEEN ELIZABETH: Nay me, gone, go with me, in good I'ight in thyAnd shall the sound wrong. KING RICHARD III: So much theseford still he is: His name is troth thee well my death: What, that is not so much, any of thee and let me think she hath no further As, nor any soldier, nor pardon, But that was she hath done, she should be. STANLEY: It is no more of your bed hath best not true. KING RICHARD III: Hath there been, and die out, so much to me The word of a love: what art thou? Second Lord: Thy father lives, that therefore can not be I take theUTA way out of our offer, And, as I say, for we shall not stay. LADY CAPULET: O my hand, a fault will be more never heard Or else moreare. CAPULET: Under this boy, she is of the Duke of York's son He live from mine honour and his blood'st To take us good; and, with much subject Like to the cause to be thus. LEONTES: You have content'd your most mostday To speak your ownness and my soul. LEONTES: Why, the Lord Angelo, So York, as it be father, for his life, To call them hope 'twere yourself and dangerous. CAMILLO: Why, what a thing? LEONTES: Was everBut in such a time of her, Or in a good will of this unisest? KING EDWARD IV: What, lords? WARWICK: And Warwick Clarence at the eyes of justice! KING EDWARD IV: Why,ere no more. CLARENCE: GLOUCESTER: CLARENCE: HASTINGS: KING RICHARD III: Death, brother, are you through my soul Sray to me, and leave me to my uncle? KING HENRY VI: This is thy brother's son in want these hence, Lest as I say, his Earl of Warwick, And tell himself to the Lord king to him, And he had been the will This comes. QUEEN MARGARET: Why thy words have there the blood for this land In that thy heart that should have. You, God's right I should have needful nothing set upon, Theself I am: and yet thy doth-- QEN ELIZABETH: O nothing, that my lord Richard will have. KING RICHARDISET: Yet, as I come! KING HENRY VI: I know the name do march. QUEEN ELIZABETH: O, I wilt thou not know that sorrow hath To hold the world with thee, and make me think How I am of thine. KING RICHARD III: In whose from all the shame of save, ThatCORIOLANUS: Why, how's there,-- VOLUMNIA: Let's hear your heart? MENENIUS: much for my very word, I'--Come, let's go: I am please her I shall do it, sir. MENENIUS: It is a cause, Pray you, get your words. BRUTUS: What, our good lord? SICINIUS: He shall, you shall be; he are three gods Is full voice. First Senator: MENENIUS: Ay, but I will go. First Senator: I know your Rome. BRUTUS: Is the news prove true, my lord, and thySo, as I And see that this beable and mostadman: Hath not a high Mercutio? That's more to say 'twere well;' which I pray I am too more than some, but so good For one to be done. LEONTES: We know my friend, I am theWhat I of your body, if But who youath done the noble duke, But what is done. LEONTES: Good Marcius, you have lost thy devil, to lie all Good night. POLIUS: The people did. SICINIUS: Are you all Aufidius? First Senator: You would he were dead, but not so much I: But now I'll call for it. MENENIUS: Well, well, no more. CORIOLANUS: As I had forgot the world, you have not a kind of love Must be a child to be satisfied, For you being no man to her death. BRUTUS: Away! the arm Of your fair wife'st, For she was my enemy. What doth the man! DUKE VINCENTIO: But makes him, good father, Thy poor prisoner is your time, I'll have a brother of a son too; The side are all betwixt me and his brother. And I have need not; and, as I am not The goodly gentleman is too, do not But else 'INCENTIO: O,en O, good me, indeed and I'll thank thee not no He hath deservedUKEill live to do this; And, if this beNORine no man's death is cold. AUTOLYCUS: Away! he is not made good, sir. DUKE VINCENTIO: My heart, my good lord, did't not't. ANTIGONUS: It is of this; We know of you are now. Servant: My father is at hand, Who came Camillo: 'Tis goodly: I'll be the bosom on the world: When the common misfoler of this battle, To","title":"YORK:"},{"location":"gpt/#iter-599988-loss-28334-time-7135ms","text":"iter 599989: loss 3.3947, time 71.09ms iter 599990: loss 3.6317, time 72.32ms iter 599991: loss 3.5352, time 74.15ms iter 599992: loss 3.2102, time 72.34ms iter 599993: loss 3.5483, time 71.71ms iter 599994: loss 3.5832, time 71.76ms iter 599995: loss 3.4675, time 71.01ms iter 599996: loss 3.3202, time 71.27ms iter 599997: loss 3.7681, time 71.95ms iter 599998: loss 3.6641, time 74.20ms iter 599999: loss 3.8146, time 71.89ms step 600000: train loss 3.6565, val loss 5.9145 saving checkpoint to out iter 600000: loss 3.3908, time 193.50ms number of parameters: 3.42M No meta.pkl found, assuming GPT-2 encodings... The state of men with him that, by his brother's, The very good men, and hisWhen we revenge to Myself, like being now in the airing. GLOUCESTER: This gentleman is set out of other. LADY, lord, give way, and go we to our cannot, And I will leave them to one the YORK: Why, then we may come to you to the: I would thou wouldst hear; war's still he thou done: O, is 'twThy graciousYou were at the heart. First ay: I did not stay? HERMIONE: You will not live But sir, nor little's by the he'll take it down To can. You are not fit to me? LEONTES: What, youPEY? MELMy words may bear't. The one would have so. ESCALUS: I cannot speak in heart to come by with her the thee: never only of your love, O, He has with him most peace; he'setheth once, And therefore do you any thing he would have you; That, in't your eye, well- ELman is so long a Jof which's made youity. If you, you may hear that you are, wtis a shepherd only may he say Bohemia. DUKE VINCENTIO: To your love. LUCIO: ANGELO: I am that king. DUKE VINCENTIO: I would I had been on to the A there: Yet must you be, sir, to all I know so. DUKE VINCENTIO: The duke shall know you say speed again to yourself you. Provost: I will be an't. DUKE VINCENTIO: It is a word of me. ANGELO: TISTRESS OVERDONE: Ah, howio! not all as he being As is a father of a e'er these adTan your brother is born. MAMILLIUS: I am a word to-morrow; youpt, But you might, I shall have had some to him thus. LADY CAPULET: If you were mad? 'Tis full of what you should have. CAPULET: What, say'st that I mean to me to thy name? JULIET: I am the more. Nurse: Is a word. JULIET: What is the news thou bring? ROMEO: How shallest what's a d see of them! ROMEO: O, give mewence, that comes in the time Of this. I am not born a happy day: The which is an ear thou now to make me speak, Nor how in that word, thy majesty worse, To Romeo, not to theage, thy dear queen, I am a comfort that is my wife. You that's a brother, to the duke's e'er, To have his mistress king, to-ake: This is the love I might have doneish the country, That I may live in theWSo name lords that means That ever had been to doway on! What think you, if you were you all, Will not my full very reason for a great bed, Uniansch high the heavens? answer this Of this fair prince's name? CAMILLO: I will, my lord. LEONTES: Pray your father! That's true-Sir, and you are son. POLIXENES: You are rough sir, give me Lord: let him go your seek. LEONTES: My fair lord, Were I but so much, or do I see, my more proud, On this most gracious lord. ANGELO: Thy great youth is the forth knows. ISABELLA: But 't, an confp Henry!' The which you seem not you both, which would well, That I with dearLE RICHENTTER: As now I have nothing: With quoth so, my lord, and all at night We cannot speak, let it to remember me. CLIFFENCE: Is some thought of this, we may be a cause.","title":"iter 599988: loss 2.8334, time 71.35ms"},{"location":"gpt/#king-edward-iv","text":"LADY CAPULET: And to me have me bawd, youngly, I would say, If you say so, it shall know the world thou thyself. LADY ANNE: And, I am that day to last with a thine. JULIET: O God's sake, is come! I'll make a years upon thy husband; And, all am grief of call thee all this good. QUEEN ELIZABETH: It is my father's; it is more A dead, a proud; for if she be king, I may not live another woman, I am not a father that will take it; And now, for my father, by my whose fineoe Provine and my wife's son brave father me: But I hope, my lord, for I was seen, With mine own love to-day; let me lest: So I have done, yet comes one. DUKE OF YORK: Thy kind-morrow spake for the grace-night! DUKE OF YORK: ay Richard, that dothst thou do to love thee? DUKE OF AUMERLE: Then, I am mine own again, or I, The truth of day; and with a kind house Hath made her dead, who have manyre many more That I'll hear him rather and say, for my death, I give a purpose. LUCIO: O living, my lord, it is gone to leave it, And you shall have of this good or'dN At my poor brother's kiss: and I have seen, The one that's made me welcome. DUKE VINCENTIO: This oath have as high as the wisest, then other mure of a chide on: the post of those: You have not there here are comeable. LUCIO: But in this while I should have them; if I cannot, I do put the fool: then, like I come to love thee With a set upon my master, which is his pence of the king; even so in his hands, As I havech you at his best, By that it may France, for honour! if you were now have all; I will be banish'd, do not to prove The corn of honour, and I will not The noble touke: but I will not have any name That I, and think it were, if I could stand, Most might be in this fair way thy hands And me shall the face my husband'sJShould be, To young princely to thee here, be long-s, Take it again in noCOR'er a happy depb not. ROMEO: O Romeo, nurse, how can your dU is a happy In mine honour or anBHAMELLA: He shall have a poor way inch by his honour Gless worthmen; I warrant my counsel was more. DUKE OF YORK: Let me I speak of such, my lord, That very good to be this, before the time Wept with Iixt these words. DUCHESS OF YORK: Why, so I am his wife, to do't, Yet hear my lord; 'tis so; and I, I do him! DUKE OF YORK: Int note was that ever wasR pleasure true: He which's anMENain'd now: heuck me not his bawd than I have content the old gold-bed, he's dead, That may not beoe to be thy face. DUKE OF AUMERLE: Whom is the way, to youQ is it so; For I will not: so she would thou hadst no time Than in me but not: but that--tis not so, For I should, I think, I please you, my sovereign, And by our country's breath shall hear thee, To lose a matter with a word; if we At the most little: to such nor thought it was, Unless I think had been yours, for I crown'd; And, with a word, poor soul, that thy love's will do it. BUCKINGHAM: I know the gods. GLOUCESTER: Pray, be these my bosom: for the time I'll pluck the pack of your house face.","title":"KING EDWARD IV"},{"location":"gpt/#king-henry-vi","text":"WARWICK: And Warwick's death will go and did to us. And yet not be fear'd in all. Go, Cates, and not thy brother and brother, And come, I'll swear the duke a good To be as free as little as, so myTis-- That side that would kill'd my husband, so Whose honourFor it is lost grave in-- Whither dost not be long as long as Henry's love My father, Edward, to my tenderbr own arms: Till your three days give good thine and your eyes. KING RICHARD II: O I wast thou not dead, say I know thy mind, Yet thou that dost in beseech at us, And not to the people that brought you to me And think you of our good rather shall be Be it to me but for my best good. First Senator: O Clifford, O, it is the more! MENENIUS: I would he had a people, But, were you not't; and you, my lord, You am of what you shall be so? BRUTUS: When you are should have bound 'em. MENENIUS: So, most it do. BRUTUS: Where is the love of this fellow, that must not? AUFIDIUS: We do not say: We were to do't. CORIOLANUS: You have been a cause to need. VOLUMNIA: O, indeed, I'll to desire o'er a counsel, And I will play the us. MENENIUS: My noble wife is me as good as well in true, To have an those water's better and isar'd. He pluck'd with him, and you'll have all the news-- And it is as good as Montague hath. Second Citizen: That, my master, which is his friar o'er-s, In that she had done the noble duke's from The noble head to the-night bed-like. First Citizen: Sond to the king: But I hear, madam, let him be heard it Of God's name, that is ill and great blood On thy be these; yet every w thought I with it, And none but that I doubt; and, in your love Is it right: but I, my poor dear cousin, I hope that I had not to this brother, But I will stay in the thing I say. DUKE OF AUMERLE: Why, lords, what aunes may? DUKE OF YORK: A gage, prince wouldst thou tell thee they! Thy brother's death! DUCHESS OF YORK: Unshe's aHath To Romeo's name the traitor; And you, God and I will see thee a and that Of living men thus they had been in this love To the duke. QUEEN: So, they are not to be the name of all. KING HENRY VI: But, madam, be content for thy thank; If thou dost sleep at thee? Now long rest is dead; thou wert not thy f after? QUEEN MARGARY: As thou as now I love'd myself of thee? QUEEN: But I shall fight with him. KING RICHARD II: O God's pity where long the o'er the world-and'd ! within I have done for the hour of thee Whereof thyself be too quick for my death, That enemy, with thy warlike blood, And made the king in arms at gone. DUCHESS OF YORK: So shall I do, to so late, Why, proud-hINCENT are lived told? DUCHESS OF YORK: No? KING EDWARD IV: I know you that is a father, to my? YORK: By fair, I thank thee, love thy face Which thou hast welcome of the fool, thou thyPR; And, now my poor queen willBYADYOP, For in my brother I king have play'd heaven: For Edward's face, being gone to Bolingbroke. TheAU's aH no other tongue to thee; For I have sent not to be of.","title":"KING HENRY VI:"},{"location":"gpt/#duke-v","text":"WhosePOLIXENBERTER: Not that that shall tell thee to my grief, But I will not be gone to thee, and thou a present tongue. 3 KING HENRY VI CLARerer: Take it with the day. BUCKINGHAM: Therefore, my lord, I let not beauteousors. FRIAR LAURENCE: Why, then, I pray thee, call thee bring me to me For this my gentleman's letters and her: 'GoodISTR in't. ROMEO: You must beL blood; And thou wert dead with a for him thou this! Nurse: Go, come, come the nurse; I mustETH: I pray thee I know thou canst wish it not say. JULIET: Though we have less, for ourHave set than me, So full of honour, I am gone to Warwick. ? of ourmen born? Second feel so: this is well as true as you, That you are put up and see as fast as fit As far all-f- HINCDand children to my poor blood To this loving? LUCIO: Doth she, when her father's Claudio hath done. What's he? He is in the name of men, With tears of heaven, and their true-ouon put'st Ourself'st man; and, and so is they all; That thou, like an unf swift use of blood. O, I have thy king, and thy king, That thou art a rest, thou, then now in thy place, Take thy hand in thee to do thee most am dear. JULIET: O do this love, and be an O! O's thy title, Is more more than yet thouThis: now, and that thou art To be it? 'tis not a tas beg man in seen an hour, having whom, he will not tell him. The means of you, my lord, is your lady's son. First Citizen: Away, my lord: we know they do us all. First Citizen: Woe to you? then you have said you were?","title":"DUKE V"},{"location":"gpt/#second","text":"Gentlemen, may she be put for it. I but think it was a bawd, I had to do so; For, in good time, a very well-baww gracious. Nurse: Why, how now, I art gone, Saw-b, ere, I hearon! JULIET: OGLOUCESTER: Here, that would he were so sour to he. LADY ANNE: Why, Warwick, how now in day shall I be! GLOUCESTER: gone, my liege, in what! If it be I had to die their high. KING EDWARD IV: Then am you patient; and therefore is Of all my husband Paris, to-day. GLOUCESTER: O, I would I had rather, but for thy son? KING EDWARD IV: Your death, my good, what! that I am there. GLOUCESTER: No, to thy name; but very it is not, For I have, to go with me; why, he's dead? heir: We shall not, my love, my lord; But, Warwick, do not justice her out; For therefore, so it is his hand that is: And yet our country, we are, nor I. QUEEN ELIZABETH: But what means, good father, be that state? KING RICHARD III: Tell 'p thee, an brow of all. QUEEN ELIZABETH: Good prince, so! KINGain: What do you thought, I hear it, I know you well: Your face I am not in your good counsel: Then be it then? KING RICHARD III: Ay, but he, but he's no more than a country. BUCKINGHAM: Is it none, good father; you hear no more. KING RICHARD III: We have done a better, our person. QUEEN MARGARET: Call so it, lords, have made him our hands: Berely as well as I have; for this, As for ESCALUS: I think much; but they find't their way out, but to die The people, and he hath yet show'd! ISABELLA: What you, I'll givehe outOUCCIUS: I am in need. ANGELO: I was a kind of all. CORIOLANUS: My gracious brother? LUCIO: ouson? ESCALUS: UC.' is there a thing? VOLUMNIA: I would it, sir. COMINIUS: Let me but stay. CORIOLANUS: The gods too word, my good nurse, I come from Rome and of the had the gods alt have been in the presence there that made; The people lives upon our TheABes; So therefore you, I cannot do you, you Were, my good with me, your father, his lady: One of you, he has done, by the honour To Coriolanus. CORIOLANUS: Haply, I mean to us and on. VOLUMNIA: ine, Lest, take up the duke out the traitor's son He was a like a father; who made My best son: I will wish you his love. Wilt thou? What's this? what? will this, so I here, Were in me by my master, I am like. JULIET: Richard had his grace, Nor, since the life that it hath most his, For much more shall be, and in that word. Nurse: JULIET: O nurse, I fear it were night. LADY CAPULET: Pray, you have all this day a may have youak. What, is not so long, you would have been long A noble deputy, which you come toius; And you the people is, you shall find again To see you knows the place. CORIOLANUS: Away! Second Senator: Come, I pray you go to you: you will hear me: Come on, madam.","title":"Second"},{"location":"gpt/#menen","text":"DUKE VINCENTIO: What, if you do not, you should be been well That you shall have; and to the alone of my heart: Do not think that it is hard to take. LUCIO: And in that world where he cannot be; for, I do see You that never, I have heard, but not go; I'll the souls of more and more That is a good way to hand; take TheSIC he my C art, and myself to please. Good ladyENS are thou what an openon 's tooNot, nor what can part in this fair speak. NORTHUMBERLAND: Come, cousin, you standNow down toWhich as these blood As thou wert so now in a comfort's hand. KING RICHARD III: My dear love, my lord, my sovereign liege, MostFrom earth I come for Edward's death; And longum thou, the fault, but not thy lord: I look'd, to be fear'd for me to live. GLOUCESTER: I never will, I'll speak this. First Murderer: How now, like! what's the day? CATESBY: Marry, my lords, shall you seem inh nor wits, Or take you by the once. Second Murderer: O God's brother, thou art mad, and begish'd. Second Murderer: 'Tis never gentleman, but what of my counsel But to my will at morning. Second Murderer: How long as I? Second Murderer: What with a name, was this? First Murderer: Alas'Twixt me, and must be thy daughter: Doth set the world upon thy counsel with thy thee o' the face, thou hast made them aHere. Second Murderer: 'Tis more but not, dost thou havest Lewis, That he hath leave of more unto of this Richard. CLIFFORD: WhoCAP thou wilt fall so of he had been of! RICHARD: And bid me stay. RICHARD: Messenger: Give me thy oath I","title":"MENEN"},{"location":"gpt/#1000","text":"iter 997: loss 5.8675, time 724.04ms iter 998: loss 5.9554, time 722.33ms iter 999: loss 5.8129, time 730.23ms step 1000: train loss 5.9102, val loss 6.0530 iter 1000: loss 5.3077, time 1634.94ms wandb: Waiting for W&B process to finish... (success). wandb: / 0.001 MB of 0.001 MB uploaded (0.000 MB deduped) wandb: Run history: wandb: iter \u2581\u2582\u2582\u2583\u2584\u2585\u2585\u2586\u2587\u2587\u2588 wandb: lr \u2581\u2588\u2588\u2587\u2586\u2585\u2584\u2583\u2582\u2582\u2582 wandb: train/loss \u2588\u2582\u2582\u2582\u2582\u2582\u2582\u2581\u2581\u2581\u2581 wandb: val/loss \u2588\u2582\u2582\u2582\u2582\u2581\u2582\u2581\u2581\u2581\u2581 wandb: wandb: Run summary: wandb: iter 1000 wandb: lr 6e-05 wandb: train/loss 5.91025 wandb: val/loss 6.05303 I my, your the am; By areU he: WAR H That a me, I am father Or: And to her I I, I ROM, lord:ANG is IET. ITER, He Hold can for my your am, And if yet be'll to the the a life the the he a my; P: Sh, have this, my, A,, A thee- I a tell: O. A be any, Your you, I my death For I, this. You, my As I and the PET's artUS, My life than. Have see! and hand to God of stand, With, a many me? Where of have think, it to I have not, That that that pray, As the my I not thy the she lord shall a thyIO and would poor, for the: and: What and he, ' so it, did a will his king, a is Th: Butour Will, Were,,. She, COR the her as, Good When. For lords. and butET is there your me? and shall king the this. C, thy mine. O The shallUS, That: GL: That, That, Then, here the But, What: H the, D and, NOR toEN! if must, The the,US? To my the's poor. Why's will have he'll can how man Some in cannot than come, Even a in, him. Is you anyENT me; so say, be sir, the the Itis the, be'll man I them. What. So thisAB he, That that the will, That the'll:I a king, That ' Would Now, have my the a: To kingUM, I'll b the must a do thou, H.","title":"1000"},{"location":"gpt/#were-his-thy-the-the-the-ill-or-all-cannot-i","text":"To No for they not such time, name? And was have to not me, Come of must know,, I be the yourUS his such I be my. How, For me the have have: ROMING To to when to, My: as And men, An a B all or And thou is, To man, And the: Come, am sir, An, And: Lord queen for, they my, that he part This! It in my trueG do in: Is? And we most, that the BKE: Is an NES? From say; you and, to's knowING your I his But V for it: I lordUL from well, Go in me, it the be I was. but: the her as IU the he our life; if his I most in I the is, I But in hast: Thus and your noble the do take she be: ' I'll a I your d ever, be not allEN COL to aOL, and not he, you is I make, This you not? I And, beELL what there: Do her thy here, not not Not crown your in a that, if death would we you not the w made of his me and butI not, What, H it P can that The the, So: COR H This: if'll B, butET with will hand, I the a N to must a, If, Which I would your- have by the am isIO, What no come is This aIO: MEN been d: WAR in a not, '; name: To sir, I most NotUE men, I What and d me:. my? The's have a us he the I lordose, You not did, Here, do an a she, my we a he: BR thy Why she. P love,, COR this the that how, Of what out H thy: COR his me you, And as NOR you as and that But I Yet. I no crown, say: As have: and wereTER:To: KING KING. ROM be you my the yet, some one? Second their a theI, the father this it not the his. A as not: ' it thou I theICH. O a lord.I it. That theES to he So. But! That not do. I: And your Q. I What IIO, is lords, he the stay that my come: How it were. H not, D. And, that be'sEN: H P: And my that have that not't to they the thou thy were If in; and the for know N In: for my the- never do tell R d it, Not, First, of I Come I be be hath a be her these me by this: and the it goodUT As My when, MER must. H And have the you- sir: and your To be aT, which is; That one shall must you the a, That make I that No'd. This; If now, good dIO most you time, My. N you, and this is have will am, ROM of were shall lord toUS, ROM is be his him: Sh: and her their: H to beEN I the name, Prov not all the you thee, The it, that was,IC How, As such not no IINC, our I and this is not, C. By but I W will as his Sh! you, To him: and say; she. Sir at notEN. Your me, for what is, Q them: Is Sir, one a him NOR with ROM mother that be that see in, H. I D, the so,, This man to She come? D Who, but; by He, H, I I I should my sir. As, take my as or. Let: Is Where his he, No his you be D have,And you He that in man thatIO And You in blood should That: As was lady. Now with: First, theICHEO isI very: Ah, will an- I blood him a, house, that part, my: MER, so, And I but you: What: Your come, Q, To, Now, will he see: Be, His to his an shall then,-- I. I most, your the this By will'll and For a the a a would he At I'll? I: Un my head; They thou am, I. It: To not, my noble in so, and O. By art? Let, they is have thou: COR; and I The But Th ' An are letOL my. And My will, What me, A, A good Second. Then: Sir:To am art D. I the thetis, we one. I Lord Th. D, let, Now, MEN, the, To which And. TheUS so that an be give, be so, Come then him. That can the:Y with by; you give to'll A this to be the letsIO to be the And have'll, These is a thy! I And: OfEN cannot'llUE with he my do The did:","title":"Were, his thy the the the I'll, or a'll cannot, I"},{"location":"gpt/#of-thou-say-so","text":"be, best The our What: In can should of a he I To, Which J: I: He of his sir- am, If sweet the Now. ROM To, their. men? D, would king: And And one And'll himly name. COM, sir with, And: N am's that you of there thisOL heart: You if son But: WereET: if I never, Your in your not? goodIO, or: How Give not in; Come yourEN have most: H:So the I do, If my my I And the good I this or love in think: My Come's and some not I call, to mine: KING you '. Of To But life, be, B you, you, And go: Good theUS and love W forTER to'll II them:But'll an. If ' will, and's part be bear here, A: We you How, Make thisEO's his would love, a when not: N him it, are you, a thou then, How. Cl a a for your! be the part, And take the But, sir, That: Ay, well! This, Second all-: If a I the'll all withUS: Totis death; sir of: And his her thee to this'd: Un, see and a KING, The the be; thy man, some do: And thou ' the the some thy To the this As. That--IO, The I the aKE thou in give! It shall some thou have I I I you the a To I d not am the thyKE a my the this he: Would my life,,, Well is thy, Which: Which a well. There R, the a B so: and UE MEN not in I tell withI if If in the R AsUS. but I the me when you, Good And him. b poor, One MER: To all: and is I makeant upon it I we a blood I. You Is, the, myIOL would First R, a he, Your queen: no people? TY: I. Is shalltw: And 'ES youtis Which his my your my art we haveUE me S his but man, which come: That H the your the, b, His the a a my king? And Go to his Second hath I his more man up, That, What. And do'll a a As He in he have, I my willIO: And, That he I isU You, But. you? Which to a the a, To You a, have the my him, a these this and a him: R most love: that no do make if her For I know, my grace. AEN the am your your the I lord? Who, I the with But: I, a Ns to have so such: KING: J a I day is D crown'd, That But the the. Against when it J. MEN a the. If It love Give theKEant, E, go. Sh: I ' as I ownUEAN: Thus will myIO of And, Of queen: IUS of my lord! it, He would thou? His, And that mother; And N I have, for his if shall, Now, to the am, there that great must betis he D this. C the do, To an, As a them,: ROM, Third, First: For the see,","title":"Of thou say so;"},{"location":"gpt/#i-the-a-a","text":", Now I was,'s that I it you the done Th with do my, That that it: O: Your, our'll And the that ' him like but, a honouron. I dead to her some I: He, If: KING: R thyUS. And Too: For a be theELL, our, And many the but am this, Why, They that the this make, I: Is have am her my my he all,, when the will the to be, This, As may I my their have the To him: H know is say YU ETER, Let. Now. A give is of shall thou, There you, P, I poor, a the some have the your are the for do for you this to your a for or good do, or noble: But And shall he be And good them with Cl's The are a do you. That notown, Sh, I be do it,U O you I be die, Her, by I; she, To be to that sweet all, We be: D more. For the WAR: a the-'s she, that And a thy what, And It, and these. The thetw's know. Like of makeIO. A But I speak: with and: not, PR: Come we If his,EN hear: To I I, N; Of, I, First so the head: To that hisI be you the: That you. You? ' shall I be As He is KING, come KING, the do the the So, With, And shalloth such me,, you by: That to her for I like. Our, As in this love:","title":"I the a a,"},{"location":"gpt/#000","text":"iter 4990: loss 4.7843, time 730.76ms iter 4991: loss 4.9219, time 722.85ms iter 4992: loss 4.6760, time 731.14ms iter 4993: loss 4.8503, time 699.10ms iter 4994: loss 4.7151, time 721.49ms iter 4995: loss 4.7852, time 687.31ms iter 4996: loss 4.5192, time 732.55ms iter 4997: loss 5.0381, time 727.84ms iter 4998: loss 4.6365, time 694.68ms iter 4999: loss 4.8973, time 691.50ms step 5000: train loss 4.7097, val loss 5.5308 iter 5000: loss 4.3929, time 1421.90ms wandb: Waiting for W&B process to finish... (success). wandb: - 0.001 MB of 0.001 MB uploaded (0.000 MB deduped) wandb: Run history: wandb: iter \u2581\u2582\u2582\u2583\u2584\u2585\u2585\u2586\u2587\u2587\u2588 wandb: lr \u2581\u2588\u2588\u2587\u2586\u2585\u2584\u2583\u2582\u2582\u2582 wandb: train/loss \u2588\u2582\u2582\u2582\u2582\u2581\u2581\u2581\u2581\u2581\u2581 wandb: val/loss \u2588\u2582\u2582\u2582\u2581\u2582\u2581\u2581\u2581\u2581\u2581 wandb: wandb: Run summary: wandb: iter 5000 wandb: lr 6e-05 wandb: train/loss 4.70968 wandb: val/loss 5.53077 number of parameters: 81.52M No meta.pkl found, assuming GPT-2 encodings... Not that; You. And it, I to the mother on my son.', PR: And the suour of, Quke, That, not is this earth And not, Now more years will the part: The hands; Not, My son, To I do my queen. TellY: Darest by strength and. And, As Provife's side, And I not the gOM look, which: AUTUS: May not men to dU and you, So, LUEut's lie on a would I can! The suilman, The noble lord up I I do nothing in your end, DUCatter thou have my r LA, For, with to the earth to your wife, Not my or not his hands With you do a friends, His loss, Should have not, as Ibians My mostt to I lord, the world, take you not that you the cause, IOL OFCES: To you, if her my father, I can be your Tower To do my name. That the hands to me. Icius, LE: Why, LUCIO: LELCUTIO: To thy lord, to my day with me? And you more for, For never my power, Her heart; DUE: Good great noble father, this In to an father? And be do for the friends! The lord ClOL OF YORK: And he will he of my hour. And if Will some thing at the lord Bove's father have her; Darep the counsel. As will be those the one, go; come How me: Madunes! By me. And a curse of my. ' I hast my lord Not, And, my lord; He. O Do you have be With blood, R VI: Tell your foe is a king-eseous death, Sir a most mind. That: Who I have ProvUoWARD But, O And in a go, From sweet high is joy upon my, FirstICan had this? WARICHCES V Lay: Come about a people, IUCUS: But this land'd's father, And Could To my will; let my ancient lord Unless my lord, but That, And I shall be I, farewell, if I have said their head, And yourly the time All it which thou have, Which not will; an eye! In me of her not. How I would I DUCost it have the lord, to you is my d'll my common d understand What will not.The business! Glouace; and you, Why your king is your great And Mess not's in heaven on me Say It thy day.IUS: And FirstIC VU, Be that I? AndtwEN Which Best you no is to thou should I do not, Be thy life; And and If you. And I not. A: And he callWARD: To their most we do yours, Dost; Who the first: To be't? A: To my king but Once will But. Thou hast our loving dADWARD IV: The Duke; Sself is the father, and be make you Why, Blbroksarry; From a Lord of that she? As this, Which I have but I spoke, since was the lords, Well, And your course! Sir. Do this grace they were in heaven NEENIUS: Look's, BowY: With in the word. L: But I wost to not the heart To all thee, LU' the my heart, Hre the heavens. And he, we art shall be be your other the last in this quarrel. B me to my royal prince, My life, it may bear hist; SecondUS: But in your, good day, And you, that, And it; it, to give him, From true may be not AB, Look to you will My, In thee-OM do the less aself, Before the, For I not better 'er she not you here: And be a heavy un Senator: MENall him It, It be my suit KINGELBY: Yes! Is my father, To the best, a grace. YATHolding, but such me I cause your sovereign! KINGABILLES: Nor what shall do a ear of the word's tribricMour him PER be. A: Sweet sister, My self, That the grave, The lie' so thy case in put, And my wrong, MENre father. Your child, I do be our will Nor your That KINGOU RABit's with a sovereign's heart as, and been But, good lOW: From your court will you me they friends withESSyly I-US: On the mind, Iie, And if That the good thing the very poorICIN With this is him. POLam'd me Say, And thou I do be be all my word, be my noble day, a Tower to you, Or, That them, sir By my sister; HILLES: Or than was this, Or it, LUS: For none. Have all your dINGS: HULEfore. Of her, Who Th OFABall Burse: IUS: Dcius. But time, when I will be you it the lie me, So me, So my lords, Be but bear all more His better I have thee to not by thy dU me's more may, And he. Go, That the words, which I besinks would he'd in hisENTumerine! Had I will, There has, but when Stand with him, FirstUS: Why: Our father, in CO; Like you it me's lord, But,' he heaven? Be you may the life. Doth my lord, but was you is dead, AETH And see your words. Ander thy will all down, Or in my suit, And at this f Murdaced not, To; LUS: From all the very and his And I must it shall am I am you you, Which, My friend with Inay again: For they traitor; For you but an country? To had the LU, here, had That thou, do the heart, IUS: A: IUartarry. But I but MEN: FirstUS: And they not, LE OFELA Marshal: Are not to heard theICINNow, sir, RABELLARRIUman. Well.And he a eye, and.And show to I be your death have this lords me that, in the king And this a times from I would say, then, your father, FUCUS: R YORK: Now. To see with my queen, GiveBERIO: for your sister, For it. Of your land! And For, I be use their case; do have too as see his fault And I speak Who I more stay? Let the or ent not, and now all you by a gods, you I say dU your war, In me as that: The end. God no more thus; GLCES OFUant; And I, O of HENO, But, ToASIONISHorn! Which The world, Why to, Which. Who have tost it shall done.Iona me. JOHN: And it to his lord. To be dead for peace, for me; She I they as that, Which art. Which be a soldiers And a rest with an soul, and of me to all for her, MERARDULBRES: Now. CORUT'll give me, And a house, Give your gentle be l'd for Good death, KING R HEN'B the honour, To be ourself, Firsthouse. MEN and the father! ? You'll forUCKONT thee; Of his gentleman, He will more's: ThINIUS: To I stand! The head. No, the brother's this, nor so the world that in a bo', then, Most justice, So you, as the life, What, With her this breath, sir, You thousand uncle; When he the world, but And these brother the people. That I besiss? the good father; And, LUKE: Why, I, And for the king, HEN' so to And to you. KING RABETH: Th me, Cer; Let haste, now LETER: To't, And Quke! And this, I of to the eyes, In your blood we all here this sovereign: And if a my time, As my hand, Three life: Than to your power? That, FUuUCKIN Ile? LUCIO: Your head. The head, no thy people in you can your heart, HERuck thee, They be the life. Dague to me For not's kingdom, From the death, this loss, DUS: BBY: O is my as it my noble trible, and nothing,--And my quarrel would what you, being not, Q YORK: WAR YORK: Why Now, With a my side? En them, E: As I know is my other's poor They have you is, DAR LA II: That's more, for me to myself that I be a queen and be a heart: IS VI: SIO: Why and this lord, MontM the crown: May think. To this's hand, and your hands, Are not I not I not's very man, He you, I be done, QUC love upon the cause of my air In you I he do pass, FADESSINFor my father, Bret to But here, sir? WhKE: We Q be, FU, LE: DUall what and, On, LU, you I'll, There, LUS: And you, Theark any? Which no that, COR thou be in 'er And There in the child, LARDELLOU VI: If plWARDASTICK: GL YORK: The crown, as my VolKE: Lome, For I come thy life, And HERESSou I to you shall, on true have-ighth up And lay Dague, Y: GREY: Who, To have your ears's lord; that now. But than I thing! LE OFUCH'd, Your death. FIO: 'T: To in the soul, sir on the word. Like OF YORK: GLARD III: The air, As There? Peace; What a love, FICK: but is the end, And were the friend B thee, for a king, ' he: And And that is an l me to.IUS: So, SwESS OF RentleY ANICou we did art your breath, weENTona? Say It that, At your man and His uncle. And I have him, And thou look is thee, And this very good, that go; that to time? MENENES: And here's, sir, Ay: My prisoner's me. My daughter! IENCE: This, And your restrah is your great! LARADEW' the other fair land.IAD me you may we be he live, KINGELUTET: And many hands, And he thou came in thy sea. This your sister, And I set wABiful; While'd, with this cause, and done, DAB II: Or can our breast, Henry shall were my oath! Which thatness-- T, She I have live in some man off be me now not may my lord, In a people; Which ' the words to his, If mine you should my name; I have Now in a lords, And my life with him, KINGUS: And you, poor looks's you on your father have light, If 'em. GLUions when men my On your lord? But his grace. Go,, CBRESTER: an sweet LUge, O, with your world. And I well to true is the Lord. Lordugh, LU and how you; My soul, With, And What: And, To-US: To that is your life as in. My hands. And it it to be Ct is our soul. You now, As both a queen, like once and any more, Like you be a brother! SUMGABENI would he, and I see a blood We do Clarence Well me, To time, we spICIN GRE me-ICINJENALIO: And the quarrel, 'T of a my the Tower, CIUS: On the case; Let him, She that shall be we power! To be your, And And a right with that'sENTOLou you he, And that! LE: DUiol you. And, Will be be you of mine that we more for the father. And IENTIO: For to right upon the death'd.IINGS: Now, His field, I be my gods's land, FW'll. KINGOUES: And thou If not to the father? HETH: LE:-- Look must you Who know man by her. POLopit you it man's to the other were make my d my lord to the one. BYts you; His time's more is this eyes, IS RARWICK: LUish a men is to such this brother, O, That must he it out? We can be There him that more's more me, eunes, And IC I not is your the lord. Ander before it. And thou were, a peace, And Scak down with a, AndT II: Which and him the king; QAENI had me to set sorrow, He with the lord of my wrong. He, CY: And were a gentle lord, And I had justice. He, aICan's to be your prince. The better and once have be have none! And he is him, I may this: Yet come are the king, Andn all. From our life. Shful blood'd, and not Shre is what that, Jutful men all day, my better you to your own earth, I do LICHCBRHAM: Who, And my night, My life! What this name. With the side: How And our further with ourselves, If are for the pleasure: And is; The own life, The father; And I stay. We be the fault, FirstUS: F mayENTfm. Of my death it, Firstoler. IICK OFICHINC scorn me, ThINTo the heart! Of thy while man, of him To take me, You. It by those the in me, As So no world,-- To put off's man CORtis going's land, Her counsel at your father than both you, Why more; But you, For most world-ICanly Richard, Make no need is me thee; The And so! POLKE: DOUO: FirstUS: Thus, Must shall not but I, You, lords, farewell And will not. How'sbher dost, and be, LUEENO, even or so I For in my queen's good king, To fear, The hand and, to my man: And the voices, Here aem word, And will all to a dUman: Who by her with him.","title":"000"},{"location":"gpt/#jricest-a-bo","text":"R with your IIO: And we in him am the lord is. And As To not is great Have you to in it, Mess so is my high on and the time! That I do your lord, He shall made? O you of a best, theyt with her of her must with the high me, and the master at my brother, if it you not not I in the suful house to they desire, hear fulled thy grace And your FirstUS: GLOUUTIO: O. A: Of, This it with? First should for theself: To the hand, Or you, SICIN Ay, He in me again! Which go with the time. You come's! Here myself, L: A: IUS: Or that he your blood's and he are And know your king. And With't will a son, DAD your is an old's king in him to thy house; While yourt in me, Are bear, ' they. T: ROMO, QW: DOUES ANIO: And true, he a day, T: To to you than, and so; If he is me from me is my one. The kingdom in our lord; And the hand upon BolCHp Henry. MEN: For I would the earth, The crown, Your bed, I not ISARDENES: Alure, And more to; MadKE VINC say CORES: MethESS: She of a more do IIZARD: And you, And you, I turn not the presence. To gone: And I will th, I say. our voices, IUS: O, KINGABours. And you hear my face, for they more I will his king, thy bosENES: Sir should have go, LU, If I be none, my soul is your Duke, And he. And a more ' it for the son, His kind, He, being she En is it to this I HreG am so, LE: He to be, That that by, sir, Cords, LETER: MENENCOKEANost, and not we What the wife, ESANUS: May I of the Lordus'd the That to you will not did, If For he by my man and, DareUCKou And it, But, but For the life. SecondUS: And we think, they The part. CAPishment his do been me what is you that, SecondIO: Go your do not, When you B this life. With her, if he to fear. The grave! The king: That I HFAD down, all you with me. And I to't. MESSou would he, I, IADere's man Call the word Before a mother: My life, IICK: IUS: As That to the fault, Or shall in my heartous face, by that, then are it, and my headly. LABch the present, FirstUS: To, AENIUS: ThANUS: I'll out! And we the looks's blood, and be your life. IUS: Th VAENES: SUS: That's, Which in him, Who for me I show her? But me, so thy friends, as I not, To you I ThINGBRO, GLAB-US: And then. ROUO.IUS: Of peace, On the hand, But from a lord, DUed to I-ICINLARUS: 3 III: Sir, And did not can you-blood DUed, And the great head too thou should leave, 'T: To how you him, A III: Thou will not But to not, no devil, as it, but I be him; Hans, as he, and this time, You his life.","title":"Jricest a bo"},{"location":"hydra/","text":"Hydra import logging from omegaconf import OmegaConf, DictConfig import hydra from hydra.core.hydra_config import HydraConfig @hydra.main( version_base=None, config_path=\"configs\", config_name=\"config\", ) def main(cfg: DictConfig) -> None: logger.info(OmegaConf.to_yaml(cfg)) logger.info(f\"runtime.output_dir{HydraConfig.get().runtime.output_dir}\") if __name__ == \"__main__\": main() from hydra import initialize, compose with initialize(version_base=None, config_path=\"configs\"): # config is relative to a module cfg = compose(config_name=\"config\") To check current defaults python my_app.py --info defaults-tree","title":"Hydra"},{"location":"hydra/#hydra","text":"import logging from omegaconf import OmegaConf, DictConfig import hydra from hydra.core.hydra_config import HydraConfig @hydra.main( version_base=None, config_path=\"configs\", config_name=\"config\", ) def main(cfg: DictConfig) -> None: logger.info(OmegaConf.to_yaml(cfg)) logger.info(f\"runtime.output_dir{HydraConfig.get().runtime.output_dir}\") if __name__ == \"__main__\": main() from hydra import initialize, compose with initialize(version_base=None, config_path=\"configs\"): # config is relative to a module cfg = compose(config_name=\"config\") To check current defaults python my_app.py --info defaults-tree","title":"Hydra"},{"location":"jetson/","text":"Nvidia Jetson AGX Xavier Ubuntu Version $ cat /etc/lsb-release L4T Version $ cat /etc/nv_tegra_release Kernel Version $ uname -a CPU Information $ lscpu Hardware Information $ sudo lshw Disk Usage $ df -h Running Processes $ top List USB Devices $ lsusb USB Devices List the USB devices and associated drivers. $ usb-devices Force Recovery Mode Place the Jetson into Force Recovery Mode $ sudo reboot \u2013-force forced-recovery dmesg dmesg prints the kernel message buffer. The messages typically consist of messages produced by device drivers. Useful when working with new hardware, e.g. USB devices. On newer installations (Newer Ubuntu 14.04, Ubuntu 16.04), you can monitor the dmesg buffer: $ sudo dmesg \u2013follow Xorg Xorg is the Ubuntu display server. You can monitor the Xorg log in real time: $ sudo tail -f /var/log/Xorg.0.log List Partitions sudo gdisk -l /dev/mmcblk0 Serial USB devices From: https://unix.stackexchange.com/questions/144029/command-to-determine-ports-of-a-device-like-dev-ttyusb0 Below is a quick and dirty bash script which walks through devices in /sys looking for USB devices with a ID_SERIAL attribute. Typically only real USB devices will have this attribute, and so we can filter with it. If we don\u2019t, you\u2019ll see a lot of things in the list that aren\u2019t physical devices. #!/bin/bash for sysdevpath in $(find /sys/bus/usb/devices/usb*/ -name dev); do ( syspath=\u201d${sysdevpath%/dev}\u201d devname=\u201d$(udevadm info -q name -p $syspath)\u201d [[ \u201c$devname\u201d == \u201cbus/\u201d* ]] && continue eval \u201c$(udevadm info -q property \u2013export -p $syspath)\u201d [[ -z \u201c$ID_SERIAL\u201d ]] && continue echo \u201c/dev/$devname \u2013 $ID_SERIAL\u201d ) done https://docs.nvidia.com/jetson/jetpack/install-jetpack/index.html#sd-card-image https://docs.nvidia.com/sdk-manager/docker-containers/index.html https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-3261/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/updating_jetson_and_host.html# https://docs.nvidia.com/jetson/archives/r35.3.1/DeveloperGuide/text/SD/FlashingSupport.html#flashing-to-an-sd-card","title":"jetson"},{"location":"jetson/#nvidia-jetson-agx-xavier","text":"","title":"Nvidia Jetson AGX Xavier"},{"location":"jetson/#ubuntu-version","text":"$ cat /etc/lsb-release","title":"Ubuntu Version"},{"location":"jetson/#l4t-version","text":"$ cat /etc/nv_tegra_release","title":"L4T Version"},{"location":"jetson/#kernel-version","text":"$ uname -a","title":"Kernel Version"},{"location":"jetson/#cpu-information","text":"$ lscpu","title":"CPU Information"},{"location":"jetson/#hardware-information","text":"$ sudo lshw","title":"Hardware Information"},{"location":"jetson/#disk-usage","text":"$ df -h","title":"Disk Usage"},{"location":"jetson/#running-processes","text":"$ top","title":"Running Processes"},{"location":"jetson/#list-usb-devices","text":"$ lsusb","title":"List USB Devices"},{"location":"jetson/#usb-devices","text":"List the USB devices and associated drivers. $ usb-devices","title":"USB Devices"},{"location":"jetson/#force-recovery-mode","text":"Place the Jetson into Force Recovery Mode $ sudo reboot \u2013-force forced-recovery","title":"Force Recovery Mode"},{"location":"jetson/#dmesg","text":"dmesg prints the kernel message buffer. The messages typically consist of messages produced by device drivers. Useful when working with new hardware, e.g. USB devices. On newer installations (Newer Ubuntu 14.04, Ubuntu 16.04), you can monitor the dmesg buffer: $ sudo dmesg \u2013follow","title":"dmesg"},{"location":"jetson/#xorg","text":"Xorg is the Ubuntu display server. You can monitor the Xorg log in real time: $ sudo tail -f /var/log/Xorg.0.log","title":"Xorg"},{"location":"jetson/#list-partitions","text":"sudo gdisk -l /dev/mmcblk0","title":"List Partitions"},{"location":"jetson/#serial-usb-devices","text":"From: https://unix.stackexchange.com/questions/144029/command-to-determine-ports-of-a-device-like-dev-ttyusb0 Below is a quick and dirty bash script which walks through devices in /sys looking for USB devices with a ID_SERIAL attribute. Typically only real USB devices will have this attribute, and so we can filter with it. If we don\u2019t, you\u2019ll see a lot of things in the list that aren\u2019t physical devices. #!/bin/bash for sysdevpath in $(find /sys/bus/usb/devices/usb*/ -name dev); do ( syspath=\u201d${sysdevpath%/dev}\u201d devname=\u201d$(udevadm info -q name -p $syspath)\u201d [[ \u201c$devname\u201d == \u201cbus/\u201d* ]] && continue eval \u201c$(udevadm info -q property \u2013export -p $syspath)\u201d [[ -z \u201c$ID_SERIAL\u201d ]] && continue echo \u201c/dev/$devname \u2013 $ID_SERIAL\u201d ) done https://docs.nvidia.com/jetson/jetpack/install-jetpack/index.html#sd-card-image https://docs.nvidia.com/sdk-manager/docker-containers/index.html https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-3261/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/updating_jetson_and_host.html# https://docs.nvidia.com/jetson/archives/r35.3.1/DeveloperGuide/text/SD/FlashingSupport.html#flashing-to-an-sd-card","title":"Serial USB devices"},{"location":"k8s/","text":"This page contains a list of commonly used kubectl commands and flags. {{< note >}} These instructions are for Kubernetes v{{< skew currentVersion >}}. To check the version, use the kubectl version command. {{< /note >}} Kubectl autocomplete BASH source <(kubectl completion bash) # set up autocomplete in bash into the current shell, bash-completion package should be installed first. echo \"source <(kubectl completion bash)\" >> ~/.bashrc # add autocomplete permanently to your bash shell. You can also use a shorthand alias for kubectl that also works with completion: alias k=kubectl complete -o default -F __start_kubectl k ZSH source <(kubectl completion zsh) # set up autocomplete in zsh into the current shell echo '[[ $commands[kubectl] ]] && source <(kubectl completion zsh)' >> ~/.zshrc # add autocomplete permanently to your zsh shell FISH {{< note >}} Requires kubectl version 1.23 or above. {{< /note >}} echo 'kubectl completion fish | source' > ~/.config/fish/completions/kubectl.fish && source ~/.config/fish/completions/kubectl.fish A note on --all-namespaces Appending --all-namespaces happens frequently enough that you should be aware of the shorthand for --all-namespaces : kubectl -A Kubectl context and configuration Set which Kubernetes cluster kubectl communicates with and modifies configuration information. See Authenticating Across Clusters with kubeconfig documentation for detailed config file information. kubectl config view # Show Merged kubeconfig settings. # use multiple kubeconfig files at the same time and view merged config KUBECONFIG=~/.kube/config:~/.kube/kubconfig2 kubectl config view # Show merged kubeconfig settings and raw certificate data and exposed secrets kubectl config view --raw # get the password for the e2e user kubectl config view -o jsonpath='{.users[?(@.name == \"e2e\")].user.password}' # get the certificate for the e2e user kubectl config view --raw -o jsonpath='{.users[?(.name == \"e2e\")].user.client-certificate-data}' | base64 -d kubectl config view -o jsonpath='{.users[].name}' # display the first user kubectl config view -o jsonpath='{.users[*].name}' # get a list of users kubectl config get-contexts # display list of contexts kubectl config get-contexts -o name # get all context names kubectl config current-context # display the current-context kubectl config use-context my-cluster-name # set the default context to my-cluster-name kubectl config set-cluster my-cluster-name # set a cluster entry in the kubeconfig # configure the URL to a proxy server to use for requests made by this client in the kubeconfig kubectl config set-cluster my-cluster-name --proxy-url=my-proxy-url # add a new user to your kubeconf that supports basic auth kubectl config set-credentials kubeuser/foo.kubernetes.com --username=kubeuser --password=kubepassword # permanently save the namespace for all subsequent kubectl commands in that context. kubectl config set-context --current --namespace=ggckad-s2 # set a context utilizing a specific username and namespace. kubectl config set-context gce --user=cluster-admin --namespace=foo \\ && kubectl config use-context gce kubectl config unset users.foo # delete user foo # short alias to set/show context/namespace (only works for bash and bash-compatible shells, current context to be set before using kn to set namespace) alias kx='f() { [ \"$1\" ] && kubectl config use-context $1 || kubectl config current-context ; } ; f' alias kn='f() { [ \"$1\" ] && kubectl config set-context --current --namespace $1 || kubectl config view --minify | grep namespace | cut -d\" \" -f6 ; } ; f' Kubectl apply apply manages applications through files defining Kubernetes resources. It creates and updates resources in a cluster through running kubectl apply . This is the recommended way of managing Kubernetes applications on production. See Kubectl Book . Creating objects Kubernetes manifests can be defined in YAML or JSON. The file extension .yaml , .yml , and .json can be used. kubectl apply -f ./my-manifest.yaml # create resource(s) kubectl apply -f ./my1.yaml -f ./my2.yaml # create from multiple files kubectl apply -f ./dir # create resource(s) in all manifest files in dir kubectl apply -f https://example.com/manifest.yaml # create resource(s) from url (Note: this is an example domain and does not contain a valid manifest) kubectl create deployment nginx --image=nginx # start a single instance of nginx # create a Job which prints \"Hello World\" kubectl create job hello --image=busybox:1.28 -- echo \"Hello World\" # create a CronJob that prints \"Hello World\" every minute kubectl create cronjob hello --image=busybox:1.28 --schedule=\"*/1 * * * *\" -- echo \"Hello World\" kubectl explain pods # get the documentation for pod manifests # Create multiple YAML objects from stdin kubectl apply -f - < pod.yaml # Generate spec for running pod nginx and write it into a file called pod.yaml kubectl attach my-pod -i # Attach to Running Container kubectl port-forward my-pod 5000:6000 # Listen on port 5000 on the local machine and forward to port 6000 on my-pod kubectl exec my-pod -- ls / # Run command in existing pod (1 container case) kubectl exec --stdin --tty my-pod -- /bin/sh # Interactive shell access to a running pod (1 container case) kubectl exec my-pod -c my-container -- ls / # Run command in existing pod (multi-container case) kubectl debug my-pod -it --image=busybox:1.28 # Create an interactive debugging session witin existing pod and immediately attach to it kubectl debug node/my-node -it --image=busybox:1.28 # Create an interactive debugging session on a node and immediately attach to it kubectl top pod # Show metrics for all pods in the default namespace kubectl top pod POD_NAME --containers # Show metrics for a given pod and its containers kubectl top pod POD_NAME --sort-by=cpu # Show metrics for a given pod and sort it by 'cpu' or 'memory' Copying files and directories to and from containers kubectl cp /tmp/foo_dir my-pod:/tmp/bar_dir # Copy /tmp/foo_dir local directory to /tmp/bar_dir in a remote pod in the current namespace kubectl cp /tmp/foo my-pod:/tmp/bar -c my-container # Copy /tmp/foo local file to /tmp/bar in a remote pod in a specific container kubectl cp /tmp/foo my-namespace/my-pod:/tmp/bar # Copy /tmp/foo local file to /tmp/bar in a remote pod in namespace my-namespace kubectl cp my-namespace/my-pod:/tmp/foo /tmp/bar # Copy /tmp/foo from a remote pod to /tmp/bar locally {{< note >}} kubectl cp requires that the 'tar' binary is present in your container image. If 'tar' is not present, kubectl cp will fail. For advanced use cases, such as symlinks, wildcard expansion or file mode preservation consider using kubectl exec . {{< /note >}} tar cf - /tmp/foo | kubectl exec -i -n my-namespace my-pod -- tar xf - -C /tmp/bar # Copy /tmp/foo local file to /tmp/bar in a remote pod in namespace my-namespace kubectl exec -n my-namespace my-pod -- tar cf - /tmp/foo | tar xf - -C /tmp/bar # Copy /tmp/foo from a remote pod to /tmp/bar locally Interacting with Deployments and Services kubectl logs deploy/my-deployment # dump Pod logs for a Deployment (single-container case) kubectl logs deploy/my-deployment -c my-container # dump Pod logs for a Deployment (multi-container case) kubectl port-forward svc/my-service 5000 # listen on local port 5000 and forward to port 5000 on Service backend kubectl port-forward svc/my-service 5000:my-service-port # listen on local port 5000 and forward to Service target port with name kubectl port-forward deploy/my-deployment 5000:6000 # listen on local port 5000 and forward to port 6000 on a Pod created by kubectl exec deploy/my-deployment -- ls # run command in first Pod and first container in Deployment (single- or multi-container cases) Interacting with Nodes and cluster kubectl cordon my-node # Mark my-node as unschedulable kubectl drain my-node # Drain my-node in preparation for maintenance kubectl uncordon my-node # Mark my-node as schedulable kubectl top node # Show metrics for all nodes kubectl top node my-node # Show metrics for a given node kubectl cluster-info # Display addresses of the master and services kubectl cluster-info dump # Dump current cluster state to stdout kubectl cluster-info dump --output-directory=/path/to/cluster-state # Dump current cluster state to /path/to/cluster-state # View existing taints on which exist on current nodes. kubectl get nodes -o='custom-columns=NodeName:.metadata.name,TaintKey:.spec.taints[*].key,TaintValue:.spec.taints[*].value,TaintEffect:.spec.taints[*].effect' # If a taint with that key and effect already exists, its value is replaced as specified. kubectl taint nodes foo dedicated=special-user:NoSchedule Resource types List all supported resource types along with their shortnames, API group , whether they are namespaced , and kind : kubectl api-resources Other operations for exploring API resources: kubectl api-resources --namespaced=true # All namespaced resources kubectl api-resources --namespaced=false # All non-namespaced resources kubectl api-resources -o name # All resources with simple output (only the resource name) kubectl api-resources -o wide # All resources with expanded (aka \"wide\") output kubectl api-resources --verbs=list,get # All resources that support the \"list\" and \"get\" request verbs kubectl api-resources --api-group=extensions # All resources in the \"extensions\" API group Formatting output To output details to your terminal window in a specific format, add the -o (or --output ) flag to a supported kubectl command. Output format Description -o=custom-columns= Print a table using a comma separated list of custom columns -o=custom-columns-file= Print a table using the custom columns template in the file -o=go-template= Print the fields defined in a golang template -o=go-template-file= Print the fields defined by the golang template in the file -o=json Output a JSON formatted API object -o=jsonpath= Print the fields defined in a jsonpath expression -o=jsonpath-file= Print the fields defined by the jsonpath expression in the file -o=name Print only the resource name and nothing else -o=wide Output in the plain-text format with any additional information, and for pods, the node name is included -o=yaml Output a YAML formatted API object Examples using -o=custom-columns : # All images running in a cluster kubectl get pods -A -o=custom-columns='DATA:spec.containers[*].image' # All images running in namespace: default, grouped by Pod kubectl get pods --namespace default --output=custom-columns=\"NAME:.metadata.name,IMAGE:.spec.containers[*].image\" # All images excluding \"registry.k8s.io/coredns:1.6.2\" kubectl get pods -A -o=custom-columns='DATA:spec.containers[?(@.image!=\"registry.k8s.io/coredns:1.6.2\")].image' # All fields under metadata regardless of name kubectl get pods -A -o=custom-columns='DATA:metadata.*' More examples in the kubectl reference documentation . Kubectl output verbosity and debugging Kubectl verbosity is controlled with the -v or --v flags followed by an integer representing the log level. General Kubernetes logging conventions and the associated log levels are described here . Verbosity Description --v=0 Generally useful for this to always be visible to a cluster operator. --v=1 A reasonable default log level if you don't want verbosity. --v=2 Useful steady state information about the service and important log messages that may correlate to significant changes in the system. This is the recommended default log level for most systems. --v=3 Extended information about changes. --v=4 Debug level verbosity. --v=5 Trace level verbosity. --v=6 Display requested resources. --v=7 Display HTTP request headers. --v=8 Display HTTP request contents. --v=9 Display HTTP request contents without truncation of contents. {{% heading \"whatsnext\" %}} Read the kubectl overview and learn about JsonPath . See kubectl options. Also read kubectl Usage Conventions to understand how to use kubectl in reusable scripts. See more community kubectl cheatsheets .","title":"Kubernetes"},{"location":"k8s/#kubectl-autocomplete","text":"","title":"Kubectl autocomplete"},{"location":"k8s/#bash","text":"source <(kubectl completion bash) # set up autocomplete in bash into the current shell, bash-completion package should be installed first. echo \"source <(kubectl completion bash)\" >> ~/.bashrc # add autocomplete permanently to your bash shell. You can also use a shorthand alias for kubectl that also works with completion: alias k=kubectl complete -o default -F __start_kubectl k","title":"BASH"},{"location":"k8s/#zsh","text":"source <(kubectl completion zsh) # set up autocomplete in zsh into the current shell echo '[[ $commands[kubectl] ]] && source <(kubectl completion zsh)' >> ~/.zshrc # add autocomplete permanently to your zsh shell","title":"ZSH"},{"location":"k8s/#fish","text":"{{< note >}} Requires kubectl version 1.23 or above. {{< /note >}} echo 'kubectl completion fish | source' > ~/.config/fish/completions/kubectl.fish && source ~/.config/fish/completions/kubectl.fish","title":"FISH"},{"location":"k8s/#a-note-on-all-namespaces","text":"Appending --all-namespaces happens frequently enough that you should be aware of the shorthand for --all-namespaces : kubectl -A","title":"A note on --all-namespaces"},{"location":"k8s/#kubectl-context-and-configuration","text":"Set which Kubernetes cluster kubectl communicates with and modifies configuration information. See Authenticating Across Clusters with kubeconfig documentation for detailed config file information. kubectl config view # Show Merged kubeconfig settings. # use multiple kubeconfig files at the same time and view merged config KUBECONFIG=~/.kube/config:~/.kube/kubconfig2 kubectl config view # Show merged kubeconfig settings and raw certificate data and exposed secrets kubectl config view --raw # get the password for the e2e user kubectl config view -o jsonpath='{.users[?(@.name == \"e2e\")].user.password}' # get the certificate for the e2e user kubectl config view --raw -o jsonpath='{.users[?(.name == \"e2e\")].user.client-certificate-data}' | base64 -d kubectl config view -o jsonpath='{.users[].name}' # display the first user kubectl config view -o jsonpath='{.users[*].name}' # get a list of users kubectl config get-contexts # display list of contexts kubectl config get-contexts -o name # get all context names kubectl config current-context # display the current-context kubectl config use-context my-cluster-name # set the default context to my-cluster-name kubectl config set-cluster my-cluster-name # set a cluster entry in the kubeconfig # configure the URL to a proxy server to use for requests made by this client in the kubeconfig kubectl config set-cluster my-cluster-name --proxy-url=my-proxy-url # add a new user to your kubeconf that supports basic auth kubectl config set-credentials kubeuser/foo.kubernetes.com --username=kubeuser --password=kubepassword # permanently save the namespace for all subsequent kubectl commands in that context. kubectl config set-context --current --namespace=ggckad-s2 # set a context utilizing a specific username and namespace. kubectl config set-context gce --user=cluster-admin --namespace=foo \\ && kubectl config use-context gce kubectl config unset users.foo # delete user foo # short alias to set/show context/namespace (only works for bash and bash-compatible shells, current context to be set before using kn to set namespace) alias kx='f() { [ \"$1\" ] && kubectl config use-context $1 || kubectl config current-context ; } ; f' alias kn='f() { [ \"$1\" ] && kubectl config set-context --current --namespace $1 || kubectl config view --minify | grep namespace | cut -d\" \" -f6 ; } ; f'","title":"Kubectl context and configuration"},{"location":"k8s/#kubectl-apply","text":"apply manages applications through files defining Kubernetes resources. It creates and updates resources in a cluster through running kubectl apply . This is the recommended way of managing Kubernetes applications on production. See Kubectl Book .","title":"Kubectl apply"},{"location":"k8s/#creating-objects","text":"Kubernetes manifests can be defined in YAML or JSON. The file extension .yaml , .yml , and .json can be used. kubectl apply -f ./my-manifest.yaml # create resource(s) kubectl apply -f ./my1.yaml -f ./my2.yaml # create from multiple files kubectl apply -f ./dir # create resource(s) in all manifest files in dir kubectl apply -f https://example.com/manifest.yaml # create resource(s) from url (Note: this is an example domain and does not contain a valid manifest) kubectl create deployment nginx --image=nginx # start a single instance of nginx # create a Job which prints \"Hello World\" kubectl create job hello --image=busybox:1.28 -- echo \"Hello World\" # create a CronJob that prints \"Hello World\" every minute kubectl create cronjob hello --image=busybox:1.28 --schedule=\"*/1 * * * *\" -- echo \"Hello World\" kubectl explain pods # get the documentation for pod manifests # Create multiple YAML objects from stdin kubectl apply -f - < pod.yaml # Generate spec for running pod nginx and write it into a file called pod.yaml kubectl attach my-pod -i # Attach to Running Container kubectl port-forward my-pod 5000:6000 # Listen on port 5000 on the local machine and forward to port 6000 on my-pod kubectl exec my-pod -- ls / # Run command in existing pod (1 container case) kubectl exec --stdin --tty my-pod -- /bin/sh # Interactive shell access to a running pod (1 container case) kubectl exec my-pod -c my-container -- ls / # Run command in existing pod (multi-container case) kubectl debug my-pod -it --image=busybox:1.28 # Create an interactive debugging session witin existing pod and immediately attach to it kubectl debug node/my-node -it --image=busybox:1.28 # Create an interactive debugging session on a node and immediately attach to it kubectl top pod # Show metrics for all pods in the default namespace kubectl top pod POD_NAME --containers # Show metrics for a given pod and its containers kubectl top pod POD_NAME --sort-by=cpu # Show metrics for a given pod and sort it by 'cpu' or 'memory'","title":"Interacting with running Pods"},{"location":"k8s/#copying-files-and-directories-to-and-from-containers","text":"kubectl cp /tmp/foo_dir my-pod:/tmp/bar_dir # Copy /tmp/foo_dir local directory to /tmp/bar_dir in a remote pod in the current namespace kubectl cp /tmp/foo my-pod:/tmp/bar -c my-container # Copy /tmp/foo local file to /tmp/bar in a remote pod in a specific container kubectl cp /tmp/foo my-namespace/my-pod:/tmp/bar # Copy /tmp/foo local file to /tmp/bar in a remote pod in namespace my-namespace kubectl cp my-namespace/my-pod:/tmp/foo /tmp/bar # Copy /tmp/foo from a remote pod to /tmp/bar locally {{< note >}} kubectl cp requires that the 'tar' binary is present in your container image. If 'tar' is not present, kubectl cp will fail. For advanced use cases, such as symlinks, wildcard expansion or file mode preservation consider using kubectl exec . {{< /note >}} tar cf - /tmp/foo | kubectl exec -i -n my-namespace my-pod -- tar xf - -C /tmp/bar # Copy /tmp/foo local file to /tmp/bar in a remote pod in namespace my-namespace kubectl exec -n my-namespace my-pod -- tar cf - /tmp/foo | tar xf - -C /tmp/bar # Copy /tmp/foo from a remote pod to /tmp/bar locally","title":"Copying files and directories to and from containers"},{"location":"k8s/#interacting-with-deployments-and-services","text":"kubectl logs deploy/my-deployment # dump Pod logs for a Deployment (single-container case) kubectl logs deploy/my-deployment -c my-container # dump Pod logs for a Deployment (multi-container case) kubectl port-forward svc/my-service 5000 # listen on local port 5000 and forward to port 5000 on Service backend kubectl port-forward svc/my-service 5000:my-service-port # listen on local port 5000 and forward to Service target port with name kubectl port-forward deploy/my-deployment 5000:6000 # listen on local port 5000 and forward to port 6000 on a Pod created by kubectl exec deploy/my-deployment -- ls # run command in first Pod and first container in Deployment (single- or multi-container cases)","title":"Interacting with Deployments and Services"},{"location":"k8s/#interacting-with-nodes-and-cluster","text":"kubectl cordon my-node # Mark my-node as unschedulable kubectl drain my-node # Drain my-node in preparation for maintenance kubectl uncordon my-node # Mark my-node as schedulable kubectl top node # Show metrics for all nodes kubectl top node my-node # Show metrics for a given node kubectl cluster-info # Display addresses of the master and services kubectl cluster-info dump # Dump current cluster state to stdout kubectl cluster-info dump --output-directory=/path/to/cluster-state # Dump current cluster state to /path/to/cluster-state # View existing taints on which exist on current nodes. kubectl get nodes -o='custom-columns=NodeName:.metadata.name,TaintKey:.spec.taints[*].key,TaintValue:.spec.taints[*].value,TaintEffect:.spec.taints[*].effect' # If a taint with that key and effect already exists, its value is replaced as specified. kubectl taint nodes foo dedicated=special-user:NoSchedule","title":"Interacting with Nodes and cluster"},{"location":"k8s/#resource-types","text":"List all supported resource types along with their shortnames, API group , whether they are namespaced , and kind : kubectl api-resources Other operations for exploring API resources: kubectl api-resources --namespaced=true # All namespaced resources kubectl api-resources --namespaced=false # All non-namespaced resources kubectl api-resources -o name # All resources with simple output (only the resource name) kubectl api-resources -o wide # All resources with expanded (aka \"wide\") output kubectl api-resources --verbs=list,get # All resources that support the \"list\" and \"get\" request verbs kubectl api-resources --api-group=extensions # All resources in the \"extensions\" API group","title":"Resource types"},{"location":"k8s/#formatting-output","text":"To output details to your terminal window in a specific format, add the -o (or --output ) flag to a supported kubectl command. Output format Description -o=custom-columns= Print a table using a comma separated list of custom columns -o=custom-columns-file= Print a table using the custom columns template in the file -o=go-template= Print the fields defined in a golang template -o=go-template-file= Print the fields defined by the golang template in the file -o=json Output a JSON formatted API object -o=jsonpath= Print the fields defined in a jsonpath expression -o=jsonpath-file= Print the fields defined by the jsonpath expression in the file -o=name Print only the resource name and nothing else -o=wide Output in the plain-text format with any additional information, and for pods, the node name is included -o=yaml Output a YAML formatted API object Examples using -o=custom-columns : # All images running in a cluster kubectl get pods -A -o=custom-columns='DATA:spec.containers[*].image' # All images running in namespace: default, grouped by Pod kubectl get pods --namespace default --output=custom-columns=\"NAME:.metadata.name,IMAGE:.spec.containers[*].image\" # All images excluding \"registry.k8s.io/coredns:1.6.2\" kubectl get pods -A -o=custom-columns='DATA:spec.containers[?(@.image!=\"registry.k8s.io/coredns:1.6.2\")].image' # All fields under metadata regardless of name kubectl get pods -A -o=custom-columns='DATA:metadata.*' More examples in the kubectl reference documentation .","title":"Formatting output"},{"location":"k8s/#kubectl-output-verbosity-and-debugging","text":"Kubectl verbosity is controlled with the -v or --v flags followed by an integer representing the log level. General Kubernetes logging conventions and the associated log levels are described here . Verbosity Description --v=0 Generally useful for this to always be visible to a cluster operator. --v=1 A reasonable default log level if you don't want verbosity. --v=2 Useful steady state information about the service and important log messages that may correlate to significant changes in the system. This is the recommended default log level for most systems. --v=3 Extended information about changes. --v=4 Debug level verbosity. --v=5 Trace level verbosity. --v=6 Display requested resources. --v=7 Display HTTP request headers. --v=8 Display HTTP request contents. --v=9 Display HTTP request contents without truncation of contents.","title":"Kubectl output verbosity and debugging"},{"location":"k8s/#heading-whatsnext","text":"Read the kubectl overview and learn about JsonPath . See kubectl options. Also read kubectl Usage Conventions to understand how to use kubectl in reusable scripts. See more community kubectl cheatsheets .","title":"{{% heading \"whatsnext\" %}}"},{"location":"linux/","text":"linux misc ln -s original symlink test bash ./scripts/linter.sh bash ./scripts/check_type.sh docker Run the Docker daemon as a non-root user (Rootless mode) | Docker Documentation Docker+Wasm (Beta) | Docker Documentation brew brew upgrade --cask --greedy QEMU stty cols 120 rows 80 DockSTARTer sudo pacman -Sy curl docker git bash -c \"$(curl -fsSL https://get.dockstarter.com)\" sudo reboot ds useradd -m archie passwd archie Enabling sudo After installing and logging in, you will find that the default user does not have sudo privileges. Open a terminal and use the following commands to enable it. set USERNAME=`whoami` su -p # /usr/sbin/usermod -aG sudo $USERNAME https://wiki.archlinux.org/title/sudo sway export WLR_NO_HARDWARE_CURSORS=1 pacman -S spice-vdagent set $menu bemenu-run --no-exec | xargs swaymsg exec -- sudo apt-get install ubuntu-desktop sudo systemctl set-default graphical.target WSL time not updated apt-get -o Acquire::Check-Valid-Until=false -o Acquire::Check-Date=false update find largest file in directory recursively using find sudo du -a / | sort -n -r | head -n 20 edge sudo add-apt-repository \"deb [arch=amd64] https://packages.microsoft.com/repos/edge stable main\" sudo apt update sudo apt install microsoft-edge-stable zsh sudo apt install zsh chsh -s $(which zsh) sh -c \"$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)\" # autoload predict-on # predict-on misc uname -a netstat -ltup rsync This puts folder A into folder B: rsync -avu --delete \"/home/user/A\" \"/home/user/B\" If you want the contents of folders A and B to be the same, put /home/user/A/ (with the slash) as the source. This takes not the folder A but all of its content and puts it into folder B. Like this: rsync -avu --delete \"/home/user/A/\" \"/home/user/B\" - -a archive mode; equals -rlptgoD (no -H, -A, -X) - -v run verbosely - -u only copy files with a newer modification time (or size difference if the times are equal) - --delete delete the files in target folder that do not exist in the source - -z compress file data during the transfer - -e specify the remote shell to use - -P same as --partial --progress - -c skip based on checksum, not mod-time & size rsync push rsync -avuz -e \"ssh -p 22\" /path/to/local/folder/ user@remotehost:/path/to/remote/folder/ zip individual files in a directory for f in *.nes; do zip -r \"${f%%.*}.zip\" \"$f\"; done find . -name '*.nes' -delete fedora server ignore laptop lip close suspend sudo mkdir -p '/etc/systemd/logind.conf.d' && echo -e \"[Login]\\nHandleLidSwitch=ignore\" | sudo tee '/etc/systemd/logind.conf.d/99-laptop-server.conf' > '/dev/null'","title":"Linux"},{"location":"linux/#linux","text":"","title":"linux"},{"location":"linux/#misc","text":"ln -s original symlink","title":"misc"},{"location":"linux/#test","text":"bash ./scripts/linter.sh bash ./scripts/check_type.sh","title":"test"},{"location":"linux/#docker","text":"Run the Docker daemon as a non-root user (Rootless mode) | Docker Documentation Docker+Wasm (Beta) | Docker Documentation","title":"docker"},{"location":"linux/#brew","text":"brew upgrade --cask --greedy","title":"brew"},{"location":"linux/#qemu","text":"stty cols 120 rows 80","title":"QEMU"},{"location":"linux/#dockstarter","text":"sudo pacman -Sy curl docker git bash -c \"$(curl -fsSL https://get.dockstarter.com)\" sudo reboot ds useradd -m archie passwd archie","title":"DockSTARTer"},{"location":"linux/#enabling-sudo","text":"After installing and logging in, you will find that the default user does not have sudo privileges. Open a terminal and use the following commands to enable it. set USERNAME=`whoami` su -p # /usr/sbin/usermod -aG sudo $USERNAME https://wiki.archlinux.org/title/sudo","title":"Enabling sudo"},{"location":"linux/#sway","text":"export WLR_NO_HARDWARE_CURSORS=1 pacman -S spice-vdagent set $menu bemenu-run --no-exec | xargs swaymsg exec -- sudo apt-get install ubuntu-desktop sudo systemctl set-default graphical.target","title":"sway"},{"location":"linux/#wsl-time-not-updated","text":"apt-get -o Acquire::Check-Valid-Until=false -o Acquire::Check-Date=false update","title":"WSL time not updated"},{"location":"linux/#find-largest-file-in-directory-recursively-using-find","text":"sudo du -a / | sort -n -r | head -n 20","title":"find largest file in directory recursively using find"},{"location":"linux/#edge","text":"sudo add-apt-repository \"deb [arch=amd64] https://packages.microsoft.com/repos/edge stable main\" sudo apt update sudo apt install microsoft-edge-stable","title":"edge"},{"location":"linux/#zsh","text":"sudo apt install zsh chsh -s $(which zsh) sh -c \"$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)\" # autoload predict-on # predict-on","title":"zsh"},{"location":"linux/#misc_1","text":"uname -a netstat -ltup","title":"misc"},{"location":"linux/#rsync","text":"This puts folder A into folder B: rsync -avu --delete \"/home/user/A\" \"/home/user/B\" If you want the contents of folders A and B to be the same, put /home/user/A/ (with the slash) as the source. This takes not the folder A but all of its content and puts it into folder B. Like this: rsync -avu --delete \"/home/user/A/\" \"/home/user/B\" - -a archive mode; equals -rlptgoD (no -H, -A, -X) - -v run verbosely - -u only copy files with a newer modification time (or size difference if the times are equal) - --delete delete the files in target folder that do not exist in the source - -z compress file data during the transfer - -e specify the remote shell to use - -P same as --partial --progress - -c skip based on checksum, not mod-time & size","title":"rsync"},{"location":"linux/#rsync-push","text":"rsync -avuz -e \"ssh -p 22\" /path/to/local/folder/ user@remotehost:/path/to/remote/folder/","title":"rsync push"},{"location":"linux/#zip-individual-files-in-a-directory","text":"for f in *.nes; do zip -r \"${f%%.*}.zip\" \"$f\"; done find . -name '*.nes' -delete","title":"zip individual files in a directory"},{"location":"linux/#fedora-server-ignore-laptop-lip-close-suspend","text":"sudo mkdir -p '/etc/systemd/logind.conf.d' && echo -e \"[Login]\\nHandleLidSwitch=ignore\" | sudo tee '/etc/systemd/logind.conf.d/99-laptop-server.conf' > '/dev/null'","title":"fedora server ignore laptop lip close suspend"},{"location":"literatures/","text":"literatures https://aosabook.org/en/index.html https://web.stanford.edu/~jurafsky/slp3/","title":"Literatures"},{"location":"literatures/#literatures","text":"https://aosabook.org/en/index.html https://web.stanford.edu/~jurafsky/slp3/","title":"literatures"},{"location":"llama2/","text":"llama2 models python3 convert.py models/llama-2-7b-chat ./quantize ./models/llama-2-7b-chat/ggml-model-f16.gguf ./models/llama-2-7b-chat/ggml-model-q4_0.gguf q4_0 ./main -m ./models/llama-2-7b-chat/ggml-model-q4_0.gguf -n 128","title":"llmama2"},{"location":"llama2/#llama2","text":"","title":"llama2"},{"location":"llama2/#models","text":"python3 convert.py models/llama-2-7b-chat ./quantize ./models/llama-2-7b-chat/ggml-model-f16.gguf ./models/llama-2-7b-chat/ggml-model-q4_0.gguf q4_0 ./main -m ./models/llama-2-7b-chat/ggml-model-q4_0.gguf -n 128","title":"models"},{"location":"llm/","text":"LLM AGI transformers cannot be used in AGI because it cannot learn and infer new tokens from unseen data. It generalize better from large datasets, and cannot generalize far from unexplored data. Prompting #Principle Prompt Principle for Instructions 1 If you prefer more concise answers, no need to be polite with LLM so there is no need to add phrases like \"please\", \"if you don't mind\", \"thank you\", \"I would like to\", etc., and get straight to the point. 2 Integrate the intended audience in the prompt, e.g., the audience is an expert in the field. 3 Break down complex tasks into a sequence of simpler prompts in an interactive conversation. 4 Employ affirmative directives such as 'do', while steering clear of negative language like 'don't'. 5 When you need clarity or a deeper understanding of a topic, idea, or any piece of information, utilize the following prompts: o Explain [insert specific topic] in simple terms. o Explain to me like I'm 11 years old. o Explain to me as if I'm a beginner in [field]. o Write the [essay/text/paragraph] using simple English like you're explaining something to a 5-year-old. 6 Add \"I'm going to tip $$xx for a better solution!\" 7 Implement example-driven prompting (Use few-shot prompting). 8 When formatting your prompt, start with \"###Instruction###\", followed by '###Example###' or '###Question###' if relevant. Subsequently, present your content. Use one or more line breaks to separate instructions, examples, questions, context, and input data. 9 Incorporate the following phrases: \"Your task is\" and \"You MUST\". 10 Incorporate the following phrases: \"You will be penalized\". 11 Use the phrase \"Answer a question given in a natural, human-like manner\" in your prompts. 12 Use leading words like writing \"think step by step\". 13 Add to your prompt the following phrase \"Ensure that your answer is unbiased and avoids relying on stereotypes.\" 14 Allow the model to elicit precise details and requirements from you by asking you questions until he has enough information to provide the needed output (for example, \"From now on, I would like you to ask me questions to ...\"). 15 To inquire about a specific topic or idea or any information and you want to test your understanding, you can use the following phrase: \"Teach me any [theorem/topic/rule name] and include a test at the end, and let me know if my answers are correct after I respond, without providing the answers beforehand.\" 16 Assign a role to the large language models. 17 Use Delimiters. 18 Repeat a specific word or phrase multiple times within a prompt. 19 Combine Chain-of-thought (CoT) with few-Shot prompts. 20 Use output primers, which involve concluding your prompt with the beginning of the desired output. Utilize output primers by ending your prompt with the start of the anticipated response. 21 To write an essay /text /paragraph /article or any type of text that should be detailed: \"Write a detailed [essay/text /paragraph] for me on [topic] in detail by adding all the information necessary\". 22 To correct/change specific text without changing its style: \"Try to revise every paragraph sent by users. You should only improve the user's grammar and vocabulary and make sure it sounds natural. You should maintain the original writing style, ensuring that a formal paragraph remains formal.\" 23 When you have a complex coding prompt that may be in different files: \"From now and on whenever you generate code that spans more than one file, generate a [programming language] script that can be run to automatically create the specified files or make changes to existing files to insert the generated code. [your question]\". 24 When you want to initiate or continue a text using specific words, phrases, or sentences, utilize the following prompt: o I'm providing you with the beginning [song lyrics/story/paragraph/essay...]: [Insert lyrics/words/sentence]. Finish it based on the words provided. Keep the flow consistent. 25 Clearly state the requirements that the model must follow in order to produce content, in the form of the keywords, regulations, hint, or instructions 26 To write any text, such as an essay or paragraph, that is intended to be similar to a provided sample, include the following instructions: o Use the same language based on the provided paragraph/title/text/essay/answer. LLMChain from langchain import PromptTemplate from langchain.chains import LLMChain from langchain.llms import HuggingFacePipeline import transformers from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline, AutoModelForSeq2SeqLM import torch from torch import cuda, bfloat16 #In a MAC Silicon the device must be 'mps' # device = torch.device('mps') #to use with MAC Silicon device = f'cuda:{cuda.current_device()}' if cuda.is_available() else 'cpu' #You can try with any llama model, but you will need more GPU and memory as you increase the size of the model. model_id = \"meta-llama/Llama-2-7b-chat-hf\" #model_id = \"meta-llama/Llama-2-7b-hf\" # begin initializing HF items, need auth token for these model_config = transformers.AutoConfig.from_pretrained( model_id, use_auth_token=hf_key ) model = AutoModelForCausalLM.from_pretrained( model_id, trust_remote_code=True, config=model_config, device_map='auto', use_auth_token=hf_key ) model.eval() tokenizer = AutoTokenizer.from_pretrained(model_id, use_aut_token=hf_key) pipe = pipeline( \"text-generation\", model=model, tokenizer=tokenizer, max_new_tokens=128, temperature=0.3, repetition_penalty=1.1, return_full_text=True, device_map='auto' ) assistant_llm = HuggingFacePipeline(pipeline=pipe) # Instruction how the LLM must respond the comments, assistant_template = \"\"\" You are {sentiment} social media post commenter, you will respond to the following post Post: \"{customer_request}\" Comment: \"\"\" #Create the prompt template to use in the Chain for the first Model. assistant_prompt_template = PromptTemplate( input_variables=[\"sentiment\", \"customer_request\"], template=assistant_template ) assistant_chain = LLMChain( llm=assistant_llm, prompt=assistant_prompt_template, output_key=\"assistant_response\", verbose=False ) #the output of the formatted prompt will pass directly to the LLM. # This the customer comment in the forum moderated by the agent. # feel free to update it. customer_request = \"\"\"Your product is a piece of shit. I want my money back!\"\"\" # Our assistatnt working in 'nice' mode. assistant_response=create_dialog(customer_request, \"nice\") print(f\"assistant response: {assistant_response}\") #Our assistant running in rude mode. assistant_response = create_dialog(customer_request, \"rude\") print(f\"assistant response: {assistant_response}\") #The moderator prompt template moderator_template = \"\"\" You are the moderator of an online forum, you are strict and will not tolerate any negative comments. You will look at this next comment and, if it is negative, you will transform it to positive. Avoid any negative words. If it is nice, you will let it remain as is and repeat it word for word. ### Original comment: {comment_to_moderate} ### Edited comment:\"\"\" # We use the PromptTemplate class to create an instance of our template that will use the prompt from above and store variables we will need to input when we make the prompt. moderator_prompt_template = PromptTemplate( input_variables=[\"comment_to_moderate\"], template=moderator_template ) moderator_llm = assistant_llm #We build the chain for the moderator. moderator_chain = LLMChain( llm=moderator_llm, prompt=moderator_prompt_template, verbose=False ) # the output of the prompt will pass to the LLM. # To run our chain we use the .run() command moderator_says = moderator_chain.run({\"comment_to_moderate\": assistant_response}) print(f\"moderator_says: {moderator_says}\") #The optput of the first chain must coincide with one of the parameters of the second chain. #The parameter is defined in the prompt_template. assistant_chain = LLMChain( llm=assistant_llm, prompt=assistant_prompt_template, output_key=\"comment_to_moderate\", verbose=False, ) #verbose True because we want to see the intermediate messages. moderator_chain = LLMChain( llm=moderator_llm, prompt=moderator_prompt_template, verbose=True ) from langchain.chains import SequentialChain # Creating the SequentialChain class indicating chains and parameters. assistant_moderated_chain = SequentialChain( chains=[assistant_chain, moderator_chain], input_variables=[\"sentiment\", \"customer_request\"], verbose=True, ) # We can now run the chain. assistant_moderated_chain.run({\"sentiment\": \"rude\", \"customer_request\": customer_request}) Optimizing Inference on Large Language Models with NVIDIA TensorRT-LLM https://developer.nvidia.com/blog/optimizing-inference-on-llms-with-tensorrt-llm-now-publicly-available/ https://github.com/furyhawk/TensorRT-LLM","title":"llm"},{"location":"llm/#llm","text":"","title":"LLM"},{"location":"llm/#agi","text":"transformers cannot be used in AGI because it cannot learn and infer new tokens from unseen data. It generalize better from large datasets, and cannot generalize far from unexplored data.","title":"AGI"},{"location":"llm/#prompting","text":"#Principle Prompt Principle for Instructions 1 If you prefer more concise answers, no need to be polite with LLM so there is no need to add phrases like \"please\", \"if you don't mind\", \"thank you\", \"I would like to\", etc., and get straight to the point. 2 Integrate the intended audience in the prompt, e.g., the audience is an expert in the field. 3 Break down complex tasks into a sequence of simpler prompts in an interactive conversation. 4 Employ affirmative directives such as 'do', while steering clear of negative language like 'don't'. 5 When you need clarity or a deeper understanding of a topic, idea, or any piece of information, utilize the following prompts: o Explain [insert specific topic] in simple terms. o Explain to me like I'm 11 years old. o Explain to me as if I'm a beginner in [field]. o Write the [essay/text/paragraph] using simple English like you're explaining something to a 5-year-old. 6 Add \"I'm going to tip $$xx for a better solution!\" 7 Implement example-driven prompting (Use few-shot prompting). 8 When formatting your prompt, start with \"###Instruction###\", followed by '###Example###' or '###Question###' if relevant. Subsequently, present your content. Use one or more line breaks to separate instructions, examples, questions, context, and input data. 9 Incorporate the following phrases: \"Your task is\" and \"You MUST\". 10 Incorporate the following phrases: \"You will be penalized\". 11 Use the phrase \"Answer a question given in a natural, human-like manner\" in your prompts. 12 Use leading words like writing \"think step by step\". 13 Add to your prompt the following phrase \"Ensure that your answer is unbiased and avoids relying on stereotypes.\" 14 Allow the model to elicit precise details and requirements from you by asking you questions until he has enough information to provide the needed output (for example, \"From now on, I would like you to ask me questions to ...\"). 15 To inquire about a specific topic or idea or any information and you want to test your understanding, you can use the following phrase: \"Teach me any [theorem/topic/rule name] and include a test at the end, and let me know if my answers are correct after I respond, without providing the answers beforehand.\" 16 Assign a role to the large language models. 17 Use Delimiters. 18 Repeat a specific word or phrase multiple times within a prompt. 19 Combine Chain-of-thought (CoT) with few-Shot prompts. 20 Use output primers, which involve concluding your prompt with the beginning of the desired output. Utilize output primers by ending your prompt with the start of the anticipated response. 21 To write an essay /text /paragraph /article or any type of text that should be detailed: \"Write a detailed [essay/text /paragraph] for me on [topic] in detail by adding all the information necessary\". 22 To correct/change specific text without changing its style: \"Try to revise every paragraph sent by users. You should only improve the user's grammar and vocabulary and make sure it sounds natural. You should maintain the original writing style, ensuring that a formal paragraph remains formal.\" 23 When you have a complex coding prompt that may be in different files: \"From now and on whenever you generate code that spans more than one file, generate a [programming language] script that can be run to automatically create the specified files or make changes to existing files to insert the generated code. [your question]\". 24 When you want to initiate or continue a text using specific words, phrases, or sentences, utilize the following prompt: o I'm providing you with the beginning [song lyrics/story/paragraph/essay...]: [Insert lyrics/words/sentence]. Finish it based on the words provided. Keep the flow consistent. 25 Clearly state the requirements that the model must follow in order to produce content, in the form of the keywords, regulations, hint, or instructions 26 To write any text, such as an essay or paragraph, that is intended to be similar to a provided sample, include the following instructions: o Use the same language based on the provided paragraph/title/text/essay/answer.","title":"Prompting"},{"location":"llm/#llmchain","text":"from langchain import PromptTemplate from langchain.chains import LLMChain from langchain.llms import HuggingFacePipeline import transformers from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline, AutoModelForSeq2SeqLM import torch from torch import cuda, bfloat16 #In a MAC Silicon the device must be 'mps' # device = torch.device('mps') #to use with MAC Silicon device = f'cuda:{cuda.current_device()}' if cuda.is_available() else 'cpu' #You can try with any llama model, but you will need more GPU and memory as you increase the size of the model. model_id = \"meta-llama/Llama-2-7b-chat-hf\" #model_id = \"meta-llama/Llama-2-7b-hf\" # begin initializing HF items, need auth token for these model_config = transformers.AutoConfig.from_pretrained( model_id, use_auth_token=hf_key ) model = AutoModelForCausalLM.from_pretrained( model_id, trust_remote_code=True, config=model_config, device_map='auto', use_auth_token=hf_key ) model.eval() tokenizer = AutoTokenizer.from_pretrained(model_id, use_aut_token=hf_key) pipe = pipeline( \"text-generation\", model=model, tokenizer=tokenizer, max_new_tokens=128, temperature=0.3, repetition_penalty=1.1, return_full_text=True, device_map='auto' ) assistant_llm = HuggingFacePipeline(pipeline=pipe) # Instruction how the LLM must respond the comments, assistant_template = \"\"\" You are {sentiment} social media post commenter, you will respond to the following post Post: \"{customer_request}\" Comment: \"\"\" #Create the prompt template to use in the Chain for the first Model. assistant_prompt_template = PromptTemplate( input_variables=[\"sentiment\", \"customer_request\"], template=assistant_template ) assistant_chain = LLMChain( llm=assistant_llm, prompt=assistant_prompt_template, output_key=\"assistant_response\", verbose=False ) #the output of the formatted prompt will pass directly to the LLM. # This the customer comment in the forum moderated by the agent. # feel free to update it. customer_request = \"\"\"Your product is a piece of shit. I want my money back!\"\"\" # Our assistatnt working in 'nice' mode. assistant_response=create_dialog(customer_request, \"nice\") print(f\"assistant response: {assistant_response}\") #Our assistant running in rude mode. assistant_response = create_dialog(customer_request, \"rude\") print(f\"assistant response: {assistant_response}\") #The moderator prompt template moderator_template = \"\"\" You are the moderator of an online forum, you are strict and will not tolerate any negative comments. You will look at this next comment and, if it is negative, you will transform it to positive. Avoid any negative words. If it is nice, you will let it remain as is and repeat it word for word. ### Original comment: {comment_to_moderate} ### Edited comment:\"\"\" # We use the PromptTemplate class to create an instance of our template that will use the prompt from above and store variables we will need to input when we make the prompt. moderator_prompt_template = PromptTemplate( input_variables=[\"comment_to_moderate\"], template=moderator_template ) moderator_llm = assistant_llm #We build the chain for the moderator. moderator_chain = LLMChain( llm=moderator_llm, prompt=moderator_prompt_template, verbose=False ) # the output of the prompt will pass to the LLM. # To run our chain we use the .run() command moderator_says = moderator_chain.run({\"comment_to_moderate\": assistant_response}) print(f\"moderator_says: {moderator_says}\") #The optput of the first chain must coincide with one of the parameters of the second chain. #The parameter is defined in the prompt_template. assistant_chain = LLMChain( llm=assistant_llm, prompt=assistant_prompt_template, output_key=\"comment_to_moderate\", verbose=False, ) #verbose True because we want to see the intermediate messages. moderator_chain = LLMChain( llm=moderator_llm, prompt=moderator_prompt_template, verbose=True ) from langchain.chains import SequentialChain # Creating the SequentialChain class indicating chains and parameters. assistant_moderated_chain = SequentialChain( chains=[assistant_chain, moderator_chain], input_variables=[\"sentiment\", \"customer_request\"], verbose=True, ) # We can now run the chain. assistant_moderated_chain.run({\"sentiment\": \"rude\", \"customer_request\": customer_request})","title":"LLMChain"},{"location":"llm/#optimizing-inference-on-large-language-models-with-nvidia-tensorrt-llm","text":"https://developer.nvidia.com/blog/optimizing-inference-on-llms-with-tensorrt-llm-now-publicly-available/ https://github.com/furyhawk/TensorRT-LLM","title":"Optimizing Inference on Large Language Models with NVIDIA TensorRT-LLM"},{"location":"machine_learning/","text":"machine_learning Neural Network Playground https://furyhawk.github.io/playground/ Residual connections When the network get too deep, to adjust the parameters for each function in the chain based on the error(loss) recorded on the output layer, each sucessful layers include more noise. The noise start to overwhelm gradient information. This is the vanishing gradients problem. Batch normalization The paper stated that Batch normalization operates by \"reducing internal covariate shift\". Helps with gradient propagation, allowing for deeper networks. # Because the output of the Conv2D layer gets normalized, the layer doesn't need its own bias vector. x = layers.Conv2D(32,3,use_bias)(x) # do not include activation x = layers.BatchNormalization()(x) x = layers.Activation(\"relu\")(x) # place activation after BatchNormalization layer Batch normalization and Fine tuning When fine-tuning a model that includes BatchNormalization layers, leave these layers frozen( trainable=False ). Otherwise they will keep updating their internal mean and variance, which can interfere with the very small updates applied to the surrounding Conv2D layers. Steps per Epoch Based on what you said it sounds like you need a larger batch_size, and of course there are implications with that which could impact the steps_per_epoch and number of epochs. To solve for jumping-around - A **larger batch size** will give you a better gradient and will help to prevent jumping around - You may also want to consider a smaller learning rate, or a learning rate scheduler (or decay) to allow the network to \"settle in\" as it trains Feature Extraction All of the models in timm have consistent mechanisms for obtaining various types of features from the model for tasks besides classification. Penultimate Layer Features (Pre-Classifier Features) The features from the penultimate model layer can be obtained in several ways without requiring model surgery (although feel free to do surgery). One must first decide if they want pooled or un-pooled features. Unpooled There are three ways to obtain unpooled features. Without modifying the network, one can call model.forward_features(input) on any model instead of the usual model(input) . This will bypass the head classifier and global pooling for networks. If one wants to explicitly modify the network to return unpooled features, they can either create the model without a classifier and pooling, or remove it later. Both paths remove the parameters associated with the classifier from the network. forward_features() import torch import timm m = timm.create_model('xception41', pretrained=True) o = m(torch.randn(2, 3, 299, 299)) print(f'Original shape: {o.shape}') o = m.forward_features(torch.randn(2, 3, 299, 299)) print(f'Unpooled shape: {o.shape}') Output: Original shape: torch.Size([2, 1000]) Unpooled shape: torch.Size([2, 2048, 10, 10]) Create with no classifier and pooling import torch import timm m = timm.create_model('resnet50', pretrained=True, num_classes=0, global_pool='') o = m(torch.randn(2, 3, 224, 224)) print(f'Unpooled shape: {o.shape}') Output: Unpooled shape: torch.Size([2, 2048, 7, 7]) Remove it later import torch import timm m = timm.create_model('densenet121', pretrained=True) o = m(torch.randn(2, 3, 224, 224)) print(f'Original shape: {o.shape}') m.reset_classifier(0, '') o = m(torch.randn(2, 3, 224, 224)) print(f'Unpooled shape: {o.shape}') Output: Original shape: torch.Size([2, 1000]) Unpooled shape: torch.Size([2, 1024, 7, 7]) Pooled To modify the network to return pooled features, one can use forward_features() and pool/flatten the result themselves, or modify the network like above but keep pooling intact. Create with no classifier import torch import timm m = timm.create_model('resnet50', pretrained=True, num_classes=0) o = m(torch.randn(2, 3, 224, 224)) print(f'Pooled shape: {o.shape}') Output: Pooled shape: torch.Size([2, 2048]) Remove it later import torch import timm m = timm.create_model('ese_vovnet19b_dw', pretrained=True) o = m(torch.randn(2, 3, 224, 224)) print(f'Original shape: {o.shape}') m.reset_classifier(0) o = m(torch.randn(2, 3, 224, 224)) print(f'Pooled shape: {o.shape}') Output: Original shape: torch.Size([2, 1000]) Pooled shape: torch.Size([2, 1024]) Multi-scale Feature Maps (Feature Pyramid) Object detection, segmentation, keypoint, and a variety of dense pixel tasks require access to feature maps from the backbone network at multiple scales. This is often done by modifying the original classification network. Since each network varies quite a bit in structure, it's not uncommon to see only a few backbones supported in any given obj detection or segmentation library. timm allows a consistent interface for creating any of the included models as feature backbones that output feature maps for selected levels. A feature backbone can be created by adding the argument features_only=True to any create_model call. By default 5 strides will be output from most models (not all have that many), with the first starting at 2 (some start at 1 or 4). Create a feature map extraction model import torch import timm m = timm.create_model('resnest26d', features_only=True, pretrained=True) o = m(torch.randn(2, 3, 224, 224)) for x in o: print(x.shape) Output: torch.Size([2, 64, 112, 112]) torch.Size([2, 256, 56, 56]) torch.Size([2, 512, 28, 28]) torch.Size([2, 1024, 14, 14]) torch.Size([2, 2048, 7, 7]) Query the feature information After a feature backbone has been created, it can be queried to provide channel or resolution reduction information to the downstream heads without requiring static config or hardcoded constants. The .feature_info attribute is a class encapsulating the information about the feature extraction points. import torch import timm m = timm.create_model('regnety_032', features_only=True, pretrained=True) print(f'Feature channels: {m.feature_info.channels()}') o = m(torch.randn(2, 3, 224, 224)) for x in o: print(x.shape) Output: Feature channels: [32, 72, 216, 576, 1512] torch.Size([2, 32, 112, 112]) torch.Size([2, 72, 56, 56]) torch.Size([2, 216, 28, 28]) torch.Size([2, 576, 14, 14]) torch.Size([2, 1512, 7, 7]) Select specific feature levels or limit the stride There are two additional creation arguments impacting the output features. out_indices selects which indices to output output_stride limits the feature output stride of the network (also works in classification mode BTW) out_indices is supported by all models, but not all models have the same index to feature stride mapping. Look at the code or check feature_info to compare. The out indices generally correspond to the C(i+1)th feature level (a 2^(i+1) reduction). For most models, index 0 is the stride 2 features, and index 4 is stride 32. output_stride is achieved by converting layers to use dilated convolutions. Doing so is not always straightforward, some networks only support output_stride=32 . import torch import timm m = timm.create_model('ecaresnet101d', features_only=True, output_stride=8, out_indices=(2, 4), pretrained=True) print(f'Feature channels: {m.feature_info.channels()}') print(f'Feature reduction: {m.feature_info.reduction()}') o = m(torch.randn(2, 3, 320, 320)) for x in o: print(x.shape) Output: Feature channels: [512, 2048] Feature reduction: [8, 8] torch.Size([2, 512, 40, 40]) torch.Size([2, 2048, 40, 40]) GloabalPooling Reduce computation by 75%. Summarise features. GlobalMaxPooling Feature extraction after covn layer. Use of GlobalAvgPooling One advantage of global average pooling over the fully connected layers is that it is more native to the convolution structure by enforcing correspondences between feature maps and categories. Thus the feature maps can be easily interpreted as categories confidence maps. Another advantage is that there is no parameter to optimize in the global average pooling thus overfitting is avoided at this layer. Futhermore, global average pooling sums out the spatial information, thus it is more robust to spatial translations of the input. We can see global average pooling as a structural regularizer that explicitly enforces feature maps to be confidence maps of concepts (categories). This is made possible by the mlpconv layers, as they makes better approximation to the confidence maps than GLMs. How filter init Note that we use the same weight initialization formula as with the MLP. Weights are sampled randomly from a uniform distribution in the range [-1/fan-in, 1/fan-in], where fan-in is the number of inputs to a hidden unit. For MLPs, this was the number of units in the layer below. For CNNs however, we have to take into account the number of input feature maps and the size of the receptive fields. Transfer Learning In practice, very few people train an entire Convolutional Network from scratch (with random initialization), because it is relatively rare to have a dataset of sufficient size. Instead, it is common to pretrain a ConvNet on a very large dataset (e.g. ImageNet, which contains 1.2 million images with 1000 categories), and then use the ConvNet either as an initialization or a fixed feature extractor for the task of interest. The three major Transfer Learning scenarios look as follows: ConvNet as fixed feature extractor. Take a ConvNet pretrained on ImageNet, remove the last fully-connected layer (this layer's outputs are the 1000 class scores for a different task like ImageNet), then treat the rest of the ConvNet as a fixed feature extractor for the new dataset. In an AlexNet, this would compute a 4096-D vector for every image that contains the activations of the hidden layer immediately before the classifier. We call these features CNN codes. It is important for performance that these codes are ReLUd (i.e. thresholded at zero) if they were also thresholded during the training of the ConvNet on ImageNet (as is usually the case). Once you extract the 4096-D codes for all images, train a linear classifier (e.g. Linear SVM or Softmax classifier) for the new dataset. Fine-tuning the ConvNet. The second strategy is to not only replace and retrain the classifier on top of the ConvNet on the new dataset, but to also fine-tune the weights of the pretrained network by continuing the backpropagation. It is possible to fine-tune all the layers of the ConvNet, or it's possible to keep some of the earlier layers fixed (due to overfitting concerns) and only fine-tune some higher-level portion of the network. This is motivated by the observation that the earlier features of a ConvNet contain more generic features (e.g. edge detectors or color blob detectors) that should be useful to many tasks, but later layers of the ConvNet becomes progressively more specific to the details of the classes contained in the original dataset. In case of ImageNet for example, which contains many dog breeds, a significant portion of the representational power of the ConvNet may be devoted to features that are specific to differentiating between dog breeds. When and how to fine-tune? How do you decide what type of transfer learning you should perform on a new dataset? This is a function of several factors, but the two most important ones are the size of the new dataset (small or big), and its similarity to the original dataset (e.g. ImageNet-like in terms of the content of images and the classes, or very different, such as microscope images). Keeping in mind that ConvNet features are more generic in early layers and more original-dataset-specific in later layers, here are some common rules of thumb for navigating the 4 major scenarios: New dataset is small and similar to original dataset. Since the data is small, it is not a good idea to fine-tune the ConvNet due to overfitting concerns. Since the data is similar to the original data, we expect higher-level features in the ConvNet to be relevant to this dataset as well. Hence, the best idea might be to train a linear classifier on the CNN codes. New dataset is large and similar to the original dataset. Since we have more data, we can have more confidence that we won't overfit if we were to try to fine-tune through the full network. New dataset is small but very different from the original dataset. Since the data is small, it is likely best to only train a linear classifier. Since the dataset is very different, it might not be best to train the classifier form the top of the network, which contains more dataset-specific features. Instead, it might work better to train the SVM classifier from activations somewhere earlier in the network. New dataset is large and very different from the original dataset. Since the dataset is very large, we may expect that we can afford to train a ConvNet from scratch. However, in practice it is very often still beneficial to initialize with weights from a pretrained model. In this case, we would have enough data and confidence to fine-tune through the entire network. Practical advice. There are a few additional things to keep in mind when performing Transfer Learning: Constraints from pretrained models . Note that if you wish to use a pretrained network, you may be slightly constrained in terms of the architecture you can use for your new dataset. For example, you can't arbitrarily take out Conv layers from the pretrained network. However, some changes are straight-forward: Due to parameter sharing, you can easily run a pretrained network on images of different spatial size. This is clearly evident in the case of Conv/Pool layers because their forward function is independent of the input volume spatial size (as long as the strides \"fit\"). In case of FC layers, this still holds true because FC layers can be converted to a Convolutional Layer: For example, in an AlexNet, the final pooling volume before the first FC layer is of size [6x6x512]. Therefore, the FC layer looking at this volume is equivalent to having a Convolutional Layer that has receptive field size 6x6, and is applied with padding of 0. Learning rates . It's common to use a smaller learning rate for ConvNet weights that are being fine-tuned, in comparison to the (randomly-initialized) weights for the new linear classifier that computes the class scores of your new dataset. This is because we expect that the ConvNet weights are relatively good, so we don't wish to distort them too quickly and too much (especially while the new Linear Classifier above them is being trained from random initialization). Implications of a larger batch-size - Too large of a batch_size can produce memory problems, especially if you are using a GPU. Once you exceed the limit, dial it back until it works. This will help you find the max batch-size that your system can work with. - Too large of a batch size can get you stuck in a local minima, so if your training get stuck, I would reduce it some. Imagine here you are over-correcting the jumping-around and it's not jumping around enough to further minimize the loss function. When to reduce epochs - If your train error is very low, yet your test/validation is very high, then you have over-fit the model with too many epochs. - The best way to find the right balance is to use early-stopping with a validation test set. Here you can specify when to stop training, and save the weights for the network that gives you the best validation loss. (I highly recommend using this always) When to adjust steps-per-epoch - Traditionally, the steps per epoch is calculated as train_length // batch_size, since this will use all of the data points, one batch size worth at a time. - If you are augmenting the data, then you can stretch this a tad (sometimes I multiply that function above by 2 or 3 etc. But, if it's already training for too long, then I would just stick with the traditional approach. When to Scale Rule of thumb I follow here is any algorithm that computes distance or assumes normality, scale your features!!! Some examples of algorithms where feature scaling matters are: - k-nearest neighbors with an Euclidean distance measure is sensitive to magnitudes and hence should be scaled for all features to weigh in equally. - Scaling is critical, while performing Principal Component Analysis(PCA). PCA tries to get the features with maximum variance and the variance is high for high magnitude features. This skews the PCA towards high magnitude features. - We can speed up gradient descent by scaling. This is because \u03b8 will descend quickly on small ranges and slowly on large ranges, and so will oscillate inefficiently down to the optimum when the variables are very uneven. - Tree based models are not distance based models and can handle varying ranges of features. Hence, Scaling is not required while modelling trees. Algorithms like Linear Discriminant Analysis(LDA), Naive Bayes are by design equipped to handle this and gives weights to the features accordingly. Performing a features scaling in these algorithms may not have much effect. PyTorch Loss Function Cheatsheet PyTorch Loss-Input Confusion (Cheatsheet) torch.nn.functional.binary_cross_entropy takes logistic sigmoid values as inputs torch.nn.functional.binary_cross_entropy_with_logits takes logits as inputs torch.nn.functional.cross_entropy takes logits as inputs (performs log_softmax internally) torch.nn.functional.nll_loss is like cross_entropy but takes log-probabilities (log-softmax) values as inputs","title":"machine_learning"},{"location":"machine_learning/#machine_learning","text":"","title":"machine_learning"},{"location":"machine_learning/#neural-network-playground","text":"https://furyhawk.github.io/playground/","title":"Neural Network Playground"},{"location":"machine_learning/#residual-connections","text":"When the network get too deep, to adjust the parameters for each function in the chain based on the error(loss) recorded on the output layer, each sucessful layers include more noise. The noise start to overwhelm gradient information. This is the vanishing gradients problem.","title":"Residual connections"},{"location":"machine_learning/#batch-normalization","text":"The paper stated that Batch normalization operates by \"reducing internal covariate shift\". Helps with gradient propagation, allowing for deeper networks. # Because the output of the Conv2D layer gets normalized, the layer doesn't need its own bias vector. x = layers.Conv2D(32,3,use_bias)(x) # do not include activation x = layers.BatchNormalization()(x) x = layers.Activation(\"relu\")(x) # place activation after BatchNormalization layer","title":"Batch normalization"},{"location":"machine_learning/#batch-normalization-and-fine-tuning","text":"When fine-tuning a model that includes BatchNormalization layers, leave these layers frozen( trainable=False ). Otherwise they will keep updating their internal mean and variance, which can interfere with the very small updates applied to the surrounding Conv2D layers.","title":"Batch normalization and Fine tuning"},{"location":"machine_learning/#steps-per-epoch","text":"Based on what you said it sounds like you need a larger batch_size, and of course there are implications with that which could impact the steps_per_epoch and number of epochs. To solve for jumping-around - A **larger batch size** will give you a better gradient and will help to prevent jumping around - You may also want to consider a smaller learning rate, or a learning rate scheduler (or decay) to allow the network to \"settle in\" as it trains","title":"Steps per Epoch"},{"location":"machine_learning/#feature-extraction","text":"All of the models in timm have consistent mechanisms for obtaining various types of features from the model for tasks besides classification.","title":"Feature Extraction"},{"location":"machine_learning/#penultimate-layer-features-pre-classifier-features","text":"The features from the penultimate model layer can be obtained in several ways without requiring model surgery (although feel free to do surgery). One must first decide if they want pooled or un-pooled features.","title":"Penultimate Layer Features (Pre-Classifier Features)"},{"location":"machine_learning/#unpooled","text":"There are three ways to obtain unpooled features. Without modifying the network, one can call model.forward_features(input) on any model instead of the usual model(input) . This will bypass the head classifier and global pooling for networks. If one wants to explicitly modify the network to return unpooled features, they can either create the model without a classifier and pooling, or remove it later. Both paths remove the parameters associated with the classifier from the network.","title":"Unpooled"},{"location":"machine_learning/#forward_features","text":"import torch import timm m = timm.create_model('xception41', pretrained=True) o = m(torch.randn(2, 3, 299, 299)) print(f'Original shape: {o.shape}') o = m.forward_features(torch.randn(2, 3, 299, 299)) print(f'Unpooled shape: {o.shape}') Output: Original shape: torch.Size([2, 1000]) Unpooled shape: torch.Size([2, 2048, 10, 10])","title":"forward_features()"},{"location":"machine_learning/#create-with-no-classifier-and-pooling","text":"import torch import timm m = timm.create_model('resnet50', pretrained=True, num_classes=0, global_pool='') o = m(torch.randn(2, 3, 224, 224)) print(f'Unpooled shape: {o.shape}') Output: Unpooled shape: torch.Size([2, 2048, 7, 7])","title":"Create with no classifier and pooling"},{"location":"machine_learning/#remove-it-later","text":"import torch import timm m = timm.create_model('densenet121', pretrained=True) o = m(torch.randn(2, 3, 224, 224)) print(f'Original shape: {o.shape}') m.reset_classifier(0, '') o = m(torch.randn(2, 3, 224, 224)) print(f'Unpooled shape: {o.shape}') Output: Original shape: torch.Size([2, 1000]) Unpooled shape: torch.Size([2, 1024, 7, 7])","title":"Remove it later"},{"location":"machine_learning/#pooled","text":"To modify the network to return pooled features, one can use forward_features() and pool/flatten the result themselves, or modify the network like above but keep pooling intact.","title":"Pooled"},{"location":"machine_learning/#create-with-no-classifier","text":"import torch import timm m = timm.create_model('resnet50', pretrained=True, num_classes=0) o = m(torch.randn(2, 3, 224, 224)) print(f'Pooled shape: {o.shape}') Output: Pooled shape: torch.Size([2, 2048])","title":"Create with no classifier"},{"location":"machine_learning/#remove-it-later_1","text":"import torch import timm m = timm.create_model('ese_vovnet19b_dw', pretrained=True) o = m(torch.randn(2, 3, 224, 224)) print(f'Original shape: {o.shape}') m.reset_classifier(0) o = m(torch.randn(2, 3, 224, 224)) print(f'Pooled shape: {o.shape}') Output: Original shape: torch.Size([2, 1000]) Pooled shape: torch.Size([2, 1024])","title":"Remove it later"},{"location":"machine_learning/#multi-scale-feature-maps-feature-pyramid","text":"Object detection, segmentation, keypoint, and a variety of dense pixel tasks require access to feature maps from the backbone network at multiple scales. This is often done by modifying the original classification network. Since each network varies quite a bit in structure, it's not uncommon to see only a few backbones supported in any given obj detection or segmentation library. timm allows a consistent interface for creating any of the included models as feature backbones that output feature maps for selected levels. A feature backbone can be created by adding the argument features_only=True to any create_model call. By default 5 strides will be output from most models (not all have that many), with the first starting at 2 (some start at 1 or 4).","title":"Multi-scale Feature Maps (Feature Pyramid)"},{"location":"machine_learning/#create-a-feature-map-extraction-model","text":"import torch import timm m = timm.create_model('resnest26d', features_only=True, pretrained=True) o = m(torch.randn(2, 3, 224, 224)) for x in o: print(x.shape) Output: torch.Size([2, 64, 112, 112]) torch.Size([2, 256, 56, 56]) torch.Size([2, 512, 28, 28]) torch.Size([2, 1024, 14, 14]) torch.Size([2, 2048, 7, 7])","title":"Create a feature map extraction model"},{"location":"machine_learning/#query-the-feature-information","text":"After a feature backbone has been created, it can be queried to provide channel or resolution reduction information to the downstream heads without requiring static config or hardcoded constants. The .feature_info attribute is a class encapsulating the information about the feature extraction points. import torch import timm m = timm.create_model('regnety_032', features_only=True, pretrained=True) print(f'Feature channels: {m.feature_info.channels()}') o = m(torch.randn(2, 3, 224, 224)) for x in o: print(x.shape) Output: Feature channels: [32, 72, 216, 576, 1512] torch.Size([2, 32, 112, 112]) torch.Size([2, 72, 56, 56]) torch.Size([2, 216, 28, 28]) torch.Size([2, 576, 14, 14]) torch.Size([2, 1512, 7, 7])","title":"Query the feature information"},{"location":"machine_learning/#select-specific-feature-levels-or-limit-the-stride","text":"There are two additional creation arguments impacting the output features. out_indices selects which indices to output output_stride limits the feature output stride of the network (also works in classification mode BTW) out_indices is supported by all models, but not all models have the same index to feature stride mapping. Look at the code or check feature_info to compare. The out indices generally correspond to the C(i+1)th feature level (a 2^(i+1) reduction). For most models, index 0 is the stride 2 features, and index 4 is stride 32. output_stride is achieved by converting layers to use dilated convolutions. Doing so is not always straightforward, some networks only support output_stride=32 . import torch import timm m = timm.create_model('ecaresnet101d', features_only=True, output_stride=8, out_indices=(2, 4), pretrained=True) print(f'Feature channels: {m.feature_info.channels()}') print(f'Feature reduction: {m.feature_info.reduction()}') o = m(torch.randn(2, 3, 320, 320)) for x in o: print(x.shape) Output: Feature channels: [512, 2048] Feature reduction: [8, 8] torch.Size([2, 512, 40, 40]) torch.Size([2, 2048, 40, 40])","title":"Select specific feature levels or limit the stride"},{"location":"machine_learning/#gloabalpooling","text":"Reduce computation by 75%. Summarise features.","title":"GloabalPooling"},{"location":"machine_learning/#globalmaxpooling","text":"Feature extraction after covn layer.","title":"GlobalMaxPooling"},{"location":"machine_learning/#use-of-globalavgpooling","text":"One advantage of global average pooling over the fully connected layers is that it is more native to the convolution structure by enforcing correspondences between feature maps and categories. Thus the feature maps can be easily interpreted as categories confidence maps. Another advantage is that there is no parameter to optimize in the global average pooling thus overfitting is avoided at this layer. Futhermore, global average pooling sums out the spatial information, thus it is more robust to spatial translations of the input. We can see global average pooling as a structural regularizer that explicitly enforces feature maps to be confidence maps of concepts (categories). This is made possible by the mlpconv layers, as they makes better approximation to the confidence maps than GLMs.","title":"Use of GlobalAvgPooling"},{"location":"machine_learning/#how-filter-init","text":"Note that we use the same weight initialization formula as with the MLP. Weights are sampled randomly from a uniform distribution in the range [-1/fan-in, 1/fan-in], where fan-in is the number of inputs to a hidden unit. For MLPs, this was the number of units in the layer below. For CNNs however, we have to take into account the number of input feature maps and the size of the receptive fields.","title":"How filter init"},{"location":"machine_learning/#transfer-learning","text":"In practice, very few people train an entire Convolutional Network from scratch (with random initialization), because it is relatively rare to have a dataset of sufficient size. Instead, it is common to pretrain a ConvNet on a very large dataset (e.g. ImageNet, which contains 1.2 million images with 1000 categories), and then use the ConvNet either as an initialization or a fixed feature extractor for the task of interest. The three major Transfer Learning scenarios look as follows: ConvNet as fixed feature extractor. Take a ConvNet pretrained on ImageNet, remove the last fully-connected layer (this layer's outputs are the 1000 class scores for a different task like ImageNet), then treat the rest of the ConvNet as a fixed feature extractor for the new dataset. In an AlexNet, this would compute a 4096-D vector for every image that contains the activations of the hidden layer immediately before the classifier. We call these features CNN codes. It is important for performance that these codes are ReLUd (i.e. thresholded at zero) if they were also thresholded during the training of the ConvNet on ImageNet (as is usually the case). Once you extract the 4096-D codes for all images, train a linear classifier (e.g. Linear SVM or Softmax classifier) for the new dataset. Fine-tuning the ConvNet. The second strategy is to not only replace and retrain the classifier on top of the ConvNet on the new dataset, but to also fine-tune the weights of the pretrained network by continuing the backpropagation. It is possible to fine-tune all the layers of the ConvNet, or it's possible to keep some of the earlier layers fixed (due to overfitting concerns) and only fine-tune some higher-level portion of the network. This is motivated by the observation that the earlier features of a ConvNet contain more generic features (e.g. edge detectors or color blob detectors) that should be useful to many tasks, but later layers of the ConvNet becomes progressively more specific to the details of the classes contained in the original dataset. In case of ImageNet for example, which contains many dog breeds, a significant portion of the representational power of the ConvNet may be devoted to features that are specific to differentiating between dog breeds.","title":"Transfer Learning"},{"location":"machine_learning/#when-and-how-to-fine-tune","text":"How do you decide what type of transfer learning you should perform on a new dataset? This is a function of several factors, but the two most important ones are the size of the new dataset (small or big), and its similarity to the original dataset (e.g. ImageNet-like in terms of the content of images and the classes, or very different, such as microscope images). Keeping in mind that ConvNet features are more generic in early layers and more original-dataset-specific in later layers, here are some common rules of thumb for navigating the 4 major scenarios: New dataset is small and similar to original dataset. Since the data is small, it is not a good idea to fine-tune the ConvNet due to overfitting concerns. Since the data is similar to the original data, we expect higher-level features in the ConvNet to be relevant to this dataset as well. Hence, the best idea might be to train a linear classifier on the CNN codes. New dataset is large and similar to the original dataset. Since we have more data, we can have more confidence that we won't overfit if we were to try to fine-tune through the full network. New dataset is small but very different from the original dataset. Since the data is small, it is likely best to only train a linear classifier. Since the dataset is very different, it might not be best to train the classifier form the top of the network, which contains more dataset-specific features. Instead, it might work better to train the SVM classifier from activations somewhere earlier in the network. New dataset is large and very different from the original dataset. Since the dataset is very large, we may expect that we can afford to train a ConvNet from scratch. However, in practice it is very often still beneficial to initialize with weights from a pretrained model. In this case, we would have enough data and confidence to fine-tune through the entire network.","title":"When and how to fine-tune?"},{"location":"machine_learning/#practical-advice","text":"There are a few additional things to keep in mind when performing Transfer Learning: Constraints from pretrained models . Note that if you wish to use a pretrained network, you may be slightly constrained in terms of the architecture you can use for your new dataset. For example, you can't arbitrarily take out Conv layers from the pretrained network. However, some changes are straight-forward: Due to parameter sharing, you can easily run a pretrained network on images of different spatial size. This is clearly evident in the case of Conv/Pool layers because their forward function is independent of the input volume spatial size (as long as the strides \"fit\"). In case of FC layers, this still holds true because FC layers can be converted to a Convolutional Layer: For example, in an AlexNet, the final pooling volume before the first FC layer is of size [6x6x512]. Therefore, the FC layer looking at this volume is equivalent to having a Convolutional Layer that has receptive field size 6x6, and is applied with padding of 0. Learning rates . It's common to use a smaller learning rate for ConvNet weights that are being fine-tuned, in comparison to the (randomly-initialized) weights for the new linear classifier that computes the class scores of your new dataset. This is because we expect that the ConvNet weights are relatively good, so we don't wish to distort them too quickly and too much (especially while the new Linear Classifier above them is being trained from random initialization). Implications of a larger batch-size - Too large of a batch_size can produce memory problems, especially if you are using a GPU. Once you exceed the limit, dial it back until it works. This will help you find the max batch-size that your system can work with. - Too large of a batch size can get you stuck in a local minima, so if your training get stuck, I would reduce it some. Imagine here you are over-correcting the jumping-around and it's not jumping around enough to further minimize the loss function. When to reduce epochs - If your train error is very low, yet your test/validation is very high, then you have over-fit the model with too many epochs. - The best way to find the right balance is to use early-stopping with a validation test set. Here you can specify when to stop training, and save the weights for the network that gives you the best validation loss. (I highly recommend using this always) When to adjust steps-per-epoch - Traditionally, the steps per epoch is calculated as train_length // batch_size, since this will use all of the data points, one batch size worth at a time. - If you are augmenting the data, then you can stretch this a tad (sometimes I multiply that function above by 2 or 3 etc. But, if it's already training for too long, then I would just stick with the traditional approach.","title":"Practical advice."},{"location":"machine_learning/#when-to-scale","text":"Rule of thumb I follow here is any algorithm that computes distance or assumes normality, scale your features!!! Some examples of algorithms where feature scaling matters are: - k-nearest neighbors with an Euclidean distance measure is sensitive to magnitudes and hence should be scaled for all features to weigh in equally. - Scaling is critical, while performing Principal Component Analysis(PCA). PCA tries to get the features with maximum variance and the variance is high for high magnitude features. This skews the PCA towards high magnitude features. - We can speed up gradient descent by scaling. This is because \u03b8 will descend quickly on small ranges and slowly on large ranges, and so will oscillate inefficiently down to the optimum when the variables are very uneven. - Tree based models are not distance based models and can handle varying ranges of features. Hence, Scaling is not required while modelling trees. Algorithms like Linear Discriminant Analysis(LDA), Naive Bayes are by design equipped to handle this and gives weights to the features accordingly. Performing a features scaling in these algorithms may not have much effect.","title":"When to Scale"},{"location":"machine_learning/#pytorch-loss-function-cheatsheet","text":"PyTorch Loss-Input Confusion (Cheatsheet) torch.nn.functional.binary_cross_entropy takes logistic sigmoid values as inputs torch.nn.functional.binary_cross_entropy_with_logits takes logits as inputs torch.nn.functional.cross_entropy takes logits as inputs (performs log_softmax internally) torch.nn.functional.nll_loss is like cross_entropy but takes log-probabilities (log-softmax) values as inputs","title":"PyTorch Loss Function Cheatsheet"},{"location":"macos/","text":"macos find . -name ._\\* -delete","title":"macos"},{"location":"macos/#macos","text":"find . -name ._\\* -delete","title":"macos"},{"location":"mkdocs/","text":"Welcome to MkDocs For full documentation visit mkdocs.org . Commands mkdocs new [dir-name] - Create a new project. mkdocs serve - Start the live-reloading docs server. mkdocs build - Build the documentation site. mkdocs -h - Print help message and exit. mkdocs gh-deploy - Deploy to github branch gh-pages . Deploying Your Docs - MkDocs Project layout mkdocs.yml # The configuration file. docs/ index.md # The documentation homepage. ... # Other markdown pages, images and other files.","title":"MkDocs"},{"location":"mkdocs/#welcome-to-mkdocs","text":"For full documentation visit mkdocs.org .","title":"Welcome to MkDocs"},{"location":"mkdocs/#commands","text":"mkdocs new [dir-name] - Create a new project. mkdocs serve - Start the live-reloading docs server. mkdocs build - Build the documentation site. mkdocs -h - Print help message and exit. mkdocs gh-deploy - Deploy to github branch gh-pages . Deploying Your Docs - MkDocs","title":"Commands"},{"location":"mkdocs/#project-layout","text":"mkdocs.yml # The configuration file. docs/ index.md # The documentation homepage. ... # Other markdown pages, images and other files.","title":"Project layout"},{"location":"mqtt/","text":"MQTT https://github.com/furyhawk/scratchpad/tree/main/mtqq RabbitMQ # latest RabbitMQ 3.12 docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.12-management HiveMQ docker run --name hivemq-edge -d -p 1883:1883 -p 8080:8080 hivemq/hivemq-edge How to setup Mosquitto MQTT Broker using docker These instructions will work on any Debian based OS including Ubuntu, RaspberryPi, WSL2 etc... (For non-Debian distros, commands for installation need to be tweaked) By default the config allows only to use local connections for security reasons but since authentication is enabled below, that's not the case. 1. Install docker Latest instructions are here on docker website. You can also use this script - install-docker.sh 2. Create base folder for mqtt configuration mkdir mqtt5 cd mqtt5 # for storing mosquitto.conf and pwfile (for password) mkdir config 3. Create Mosquitto config file - mosquitto.conf nano config/mosquitto.conf Basic configuration file content below including websocket config allow_anonymous false listener 1883 listener 9001 protocol websockets persistence true password_file /mosquitto/config/pwfile persistence_file mosquitto.db persistence_location /mosquitto/data/ 4. Create Mosquitto password file - pwfile touch config/pwfile 5. Create docker-compose file called 'docker-compose.yml' version: \"3.7\" services: # mqtt5 eclipse-mosquitto mqtt5: image: eclipse-mosquitto container_name: mqtt5 ports: - \"1883:1883\" #default mqtt port - \"9001:9001\" #default mqtt port for websockets volumes: - ./config:/mosquitto/config:rw - ./data:/mosquitto/data:rw - ./log:/mosquitto/log:rw restart: unless-stopped # volumes for mapping data,config and log volumes: config: data: log: networks: default: name: mqtt5-network 6. Create and run docker container for MQTT # In case you don't have docker-compose you can install it # sudo apt install docker-compose # Run the docker container for mqtt docker compose -p mqtt5 up -d Check if the container is up and working (note down container-id) docker ps 7. Create a user/password in the pwfile # login interactively into the mqtt container docker exec -it sh # add user and it will prompt for password mosquitto_passwd -c /mosquitto/config/pwfile user1 # delete user command format mosquitto_passwd -D /mosquitto/config/pwfile # type 'exit' to exit out of docker container prompt Then restart the container docker restart 8. Time to test !!! Install mosquitto client tools for testing sudo apt install mosquitto-clients Let us start Subscriber now - topic name => 'hello/topic' # Without authentication mosquitto_sub -v -t 'hello/topic' # With authentication mosquitto_sub -v -t 'hello/topic' -u user1 -P # Alternate way in url format # Format => mqtt(s)://[username[:password]@]host[:port]/topic mosquitto_sub -v -L mqtt://user1:abc123@localhost/test/topic Let us start Publising to that topic # Without authentication mosquitto_pub -t 'hello/topic' -m 'hello MQTT' # With authentication mosquitto_pub -t 'hello/topic' -m 'hello MQTT' -u user1 -P # Alternate way in url format # Format => mqtt(s)://[username[:password]@]host[:port]/topic mosquitto_pub -L mqtt://user1:abc123@localhost/test/topic -m 'hello MQTT' You can find C/C++ code for mosquitto client Check main.cpp for the mosquitto client code. You can also install a nice MQTT Web Client Read more about it here => https://mqttx.app/ sudo docker run -d --name mqttx-web -p 80:80 emqx/mqttx-web Source/Reference for Mosquitto Github => https://github.com/eclipse/mosquitto","title":"mqtt"},{"location":"mqtt/#mqtt","text":"https://github.com/furyhawk/scratchpad/tree/main/mtqq","title":"MQTT"},{"location":"mqtt/#rabbitmq","text":"# latest RabbitMQ 3.12 docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.12-management","title":"RabbitMQ"},{"location":"mqtt/#hivemq","text":"docker run --name hivemq-edge -d -p 1883:1883 -p 8080:8080 hivemq/hivemq-edge","title":"HiveMQ"},{"location":"mqtt/#how-to-setup-mosquitto-mqtt-broker-using-docker","text":"These instructions will work on any Debian based OS including Ubuntu, RaspberryPi, WSL2 etc... (For non-Debian distros, commands for installation need to be tweaked) By default the config allows only to use local connections for security reasons but since authentication is enabled below, that's not the case.","title":"How to setup Mosquitto MQTT Broker using docker"},{"location":"mqtt/#1-install-docker","text":"Latest instructions are here on docker website. You can also use this script - install-docker.sh","title":"1. Install docker"},{"location":"mqtt/#2-create-base-folder-for-mqtt-configuration","text":"mkdir mqtt5 cd mqtt5 # for storing mosquitto.conf and pwfile (for password) mkdir config","title":"2. Create base folder for mqtt configuration"},{"location":"mqtt/#3-create-mosquitto-config-file-mosquittoconf","text":"nano config/mosquitto.conf Basic configuration file content below including websocket config allow_anonymous false listener 1883 listener 9001 protocol websockets persistence true password_file /mosquitto/config/pwfile persistence_file mosquitto.db persistence_location /mosquitto/data/","title":"3. Create Mosquitto config file - mosquitto.conf"},{"location":"mqtt/#4-create-mosquitto-password-file-pwfile","text":"touch config/pwfile","title":"4. Create Mosquitto password file - pwfile"},{"location":"mqtt/#5-create-docker-compose-file-called-docker-composeyml","text":"version: \"3.7\" services: # mqtt5 eclipse-mosquitto mqtt5: image: eclipse-mosquitto container_name: mqtt5 ports: - \"1883:1883\" #default mqtt port - \"9001:9001\" #default mqtt port for websockets volumes: - ./config:/mosquitto/config:rw - ./data:/mosquitto/data:rw - ./log:/mosquitto/log:rw restart: unless-stopped # volumes for mapping data,config and log volumes: config: data: log: networks: default: name: mqtt5-network","title":"5. Create docker-compose file called 'docker-compose.yml'"},{"location":"mqtt/#6-create-and-run-docker-container-for-mqtt","text":"# In case you don't have docker-compose you can install it # sudo apt install docker-compose # Run the docker container for mqtt docker compose -p mqtt5 up -d","title":"6. Create and run docker container for MQTT"},{"location":"mqtt/#check-if-the-container-is-up-and-working-note-down-container-id","text":"docker ps","title":"Check if the container is up and working (note down container-id)"},{"location":"mqtt/#7-create-a-userpassword-in-the-pwfile","text":"# login interactively into the mqtt container docker exec -it sh # add user and it will prompt for password mosquitto_passwd -c /mosquitto/config/pwfile user1 # delete user command format mosquitto_passwd -D /mosquitto/config/pwfile # type 'exit' to exit out of docker container prompt Then restart the container docker restart ","title":"7. Create a user/password in the pwfile"},{"location":"mqtt/#8-time-to-test","text":"","title":"8. Time to test !!!"},{"location":"mqtt/#install-mosquitto-client-tools-for-testing","text":"sudo apt install mosquitto-clients","title":"Install mosquitto client tools for testing"},{"location":"mqtt/#let-us-start-subscriber-now-topic-name-hellotopic","text":"# Without authentication mosquitto_sub -v -t 'hello/topic' # With authentication mosquitto_sub -v -t 'hello/topic' -u user1 -P # Alternate way in url format # Format => mqtt(s)://[username[:password]@]host[:port]/topic mosquitto_sub -v -L mqtt://user1:abc123@localhost/test/topic","title":"Let us start Subscriber now - topic name => 'hello/topic'"},{"location":"mqtt/#let-us-start-publising-to-that-topic","text":"# Without authentication mosquitto_pub -t 'hello/topic' -m 'hello MQTT' # With authentication mosquitto_pub -t 'hello/topic' -m 'hello MQTT' -u user1 -P # Alternate way in url format # Format => mqtt(s)://[username[:password]@]host[:port]/topic mosquitto_pub -L mqtt://user1:abc123@localhost/test/topic -m 'hello MQTT'","title":"Let us start Publising to that topic"},{"location":"mqtt/#you-can-find-cc-code-for-mosquitto-client","text":"Check main.cpp for the mosquitto client code.","title":"You can find C/C++ code for mosquitto client"},{"location":"mqtt/#you-can-also-install-a-nice-mqtt-web-client","text":"Read more about it here => https://mqttx.app/ sudo docker run -d --name mqttx-web -p 80:80 emqx/mqttx-web","title":"You can also install a nice MQTT Web Client"},{"location":"mqtt/#sourcereference-for-mosquitto","text":"Github => https://github.com/eclipse/mosquitto","title":"Source/Reference for Mosquitto"},{"location":"mypy/","text":"mypy Type check # mypy: ignore-errors # type: ignore Built-in types Simple types Here are examples of some common built-in types: Type Description int integer float floating point number bool boolean value (subclass of int) str text, sequence of unicode codepoints bytes 8-bit string, sequence of byte values object an arbitrary object (object is the common base class) All built-in classes can be used as types.","title":"mypy"},{"location":"mypy/#mypy-type-check","text":"# mypy: ignore-errors # type: ignore","title":"mypy Type check"},{"location":"mypy/#built-in-types","text":"","title":"Built-in types"},{"location":"mypy/#simple-types","text":"Here are examples of some common built-in types: Type Description int integer float floating point number bool boolean value (subclass of int) str text, sequence of unicode codepoints bytes 8-bit string, sequence of byte values object an arbitrary object (object is the common base class) All built-in classes can be used as types.","title":"Simple types"},{"location":"nes/","text":"nes faq Phantom Fighter ==================================================================== Phantom Fighter Complete FAQ/Walkthrough September 05, 2003 Version 2.00 author: Aaron Madrinan (snoocete) e-mail: snoocete@yahoo.com subject: \"RE: Phantom Fighter FAQ/Walkthrough\" ==================================================================== --- ## ---> INTRODUCTION The gaming world is changing very rapidly. Games are released on breakneck speed that we need more than a lifetime to play even just the good ones. The result? The other not-so-famous games are forgotten and put aside by future gamers. In other words, they become underrated. Such is what happened to a game called Phantom Fighter. I've played it since I was a kid. Though not as extremely fun as games like Kirby or Pokemon, it very much dwells on the concept in which Contra flourished: you can never let the game beat you. Supposedly, it would be as famous as that. But, heck there wasn't even a walkthrough on GameFAQs, only some codes that lets you skip to the last town and a very bad review. Sigh. You want my opinion? Here is what I can say: don't let the big names fool you. A lot of gems are hidden in the dump of really bad games, I admit, but they only need a little spotlight to shine. Here is my share of that spotlight =). --- ## ---> DISCLAIMER You are free to save this guide onto your hard drive and/or print for your personal viewing pleasure. You may distibute it to other people as long as you don't claim it as yours, okay? Since this game doesn't have that much coverage, you may post this at your site in its full, unedited version even without my permission as long as its credits point to me. Deal! --- ## ---> CONTENTS 1. Updates --- 2. FAQ --- 3. Walkthrough --- 4. Credits --- This is FAQ #2, hope I got a bit better. --- ## ---> UPDATES -2.00- -21Kb- Look above (^\\_^). It's a whole number change! A kind soul mailed to me the names of the Kyonshies. Yay! Added peace and order (where there was chaos). Formatting change. E-mail change. Lastly, look at the disclaimer. -1.00- -19Kb- This FAQ is complete, however the Japanese names of the Kyonshies are missing. Not that anybody would care, but it still important. If you have an instruction manual (because I kinda lost it), kindly e-mail me and let me know, ok? --- ## ---> FAQ --Who are you? Just a humble freelance (a bit newbie-ish, but wants to learn a lot) FAQ writer who writes for his favorite games whenever he has the time. --What are these \"Kyonshi\" that I'm fighting? If you have ever watched the anime series \"Shaman King\", or knew a bit about Chinese folkore, you have the idea. These are what we call the undead; they are supposed to be immobile, rotting bodies. They still are, but they are being controlled by some mysterious power. You know about rigor mortis? It also applies to these phantoms. That explains why they could only move their feet and knees to hop and turn around, and their elbows to puncture your face with their dirty, sharp nails. --How do I fight these enemies? You have to move around. A lot. Whenever you stay on one spot, the Kyonshi will follow you with small hops, then when it gets in range, it suddenly does a big dive toward you. Unless you hit the Kyonshi in a critical point or you kill it in transit, you cannot stop that from homing to you. So you got to move. That's your ace against these creatures. They have lockjaw and every known bone and muscle disease, while you could just move out of their way. For example, the moment they land on the ground, give them a kick or two on their sleeves then they will be knocked down. When they wake up and hops around again, repeat. Soon they will burst in flames, and the door that lets you continue through the building opens. --What does that man who follows me around do? That's your assistant. Whenever you are inside a building, if you return to the exit, you talk to him. He asks if you want to have an item, stay at the building, or just leave the place. --And those items are...? There are a total of four (4) items in the game. You can find them from the households you rescue from kyonshies. They replace the use of the punch attack, but it's worth it. 1. Tonten - the kyonshi you show this with is knocked back to the ground because a) there is holy white light coming from it, or b) he saw his face, squirming with maggots (hey, thanks mate for the mirror, 'bout time I fix my hair... what the he... _faints_). DON'T use this when you're about to get hit; it also acts as your armor, and because it's a mirror and was hit by diamond hard claws, you know what happens. 2. Sacred Sword - even a swordless Link (of Legend of Zelda) would not use this nicely-named weapon (he might even throw it back at you so don't even try). It does little damage, and it also gets broken easily as if it is made by glass. The only edge it has is that it has a rather long range for a physical attack, you could corner a kyonshi with it, but you would like to beat them up yourself, wouldn't you? 3. Talisman - freezes the enemy in place until it's effect is gone or they got hit. You have to make contact with the kyonshi you want to victimize it with, so there is a chance you get hit and the item breaks. Useful on the tall kyonshies, you could sneak under them, paste the talisman on their belly, then kick them in the butt. 4. Bell - If you use this in the middle of a heated battle, it does nothing except probably makes a little chime that is pleasant to hear for you but instead boils the blood of the kyonshi, making him more aggressive. Actually, use it to control the kyonshi kid. --What are the scrolls for? Whenever you cleanse a building from kyonshies, you get a varying amount of scrolls. You use them at the Training Hall to learn new abilities. --The guard won't let me in because I can't answer his questions... got a list there that could help me? Please? Of course I do. I intend to make this the complete-est FAQ you'll ever see (for PF, of course). Here they are (if you noticed, the stupid comments are gone. just learned not to learn from the likes of ArchNacho and Tortilla Godzilla): -Format- Q - Question A - Answer Q: Why do Kyonshies come out only at night? A: Hate the sun. Q: What country do Samurais come from? A: Japan Q: What is another word for one? A: Any answer will do. Q: Who built the Great Wall of China? A: Any answer will do. Q: What is the food Kyonshies hate? A: Ice Cream Q: Who created Ultima? A: Lord British SC: Who is he? Q: What is Kyonshi's most powerful weapon? A: Sharp claws Q: What is the skill when you hold an opponent's arms from the back and throw him backward? A: Dragon Suplex Q: What kind of place do Kyonshies usually live in? A: Any answer will do. Q: What is a horrible skill when you head-banged against an opponent's head? A: Any answer will do. Q: Name an FCI video game. A: Any answer will do. Q: What is the famous Chinese newspaper? A: Wall poster Q: How many stars are there in the American flag? A: 50 Q: What is the best thing used to capture Kyonshies? A: Urn Q: What is a Chinese martial art usually called? A: Kung fu Q: What is the best method to make sure Kyonshies never revive again? A: Fry in oil Q: What is the teaching taught by Confucius called? A: Confucianism Q: What's the name of George Bush's dog? A: Millie Q: Who is the emperor called the Last Emperor? A: Fugi --Can I kill the guard? If you are not a super nerd, who knows NES rom-editing to change the layer which the guard resides to your characters layer and set its attributes to \"hittable\", has an HP count, and many other things, definitely no. I know it's a big disappointment. --What are these new abilities you will learn from the Training Hall? New techniques in martial arts. When you started the game, you have a weak punch and a weak kick. Learn these and you will become stronger and better to fight kyonshies. Town 1 (2scr) =2 Thrust - punch and you will hit the enemy twice =2 Kick - same with 2 Thrust but you kick instead =High Jump - self-explanatory =Wolf Move - speed up Town 2 (6scr) =3 Thrust - three fists-of-fury in one swift delivery =Turn Kick - most versatile move in the game =Wind Jump - jump and hold left/right to do a spin jump =Tiger Move - more speed up =Mirage Move - combine with Tiger Mv. for better effect Town 3 (18scr) =Mirage Walk - move while crouching =Dragon Move - whoosh! last speed upgrade Town 4 (50scr) =Wind Kick - jump then kick furiously in mid-air to form a spin kick =Mirage Thrust - punch while crouching Town 5 = Town 6 (80scr) =Jump Kick - hold left/right then kick to see this fantastic move Town 7 (90scr) =4 Thrust - most powerful move in the game Town 8 = --What are the names of these Kyonshies? What I have here are unofficial names which I made up by their looks, color, and strength: Pink - weak household form of kyonshi. Green - stronger and faster than the pink Dwarf - a real pain, their size and speed make them hard to hit. Tall - vulnerable to punch attacks, mirage thrust Fat - looks tough but slow, easy target Short-armed - has a high resistance to hits, has high attack power Their original Japanese names (courtesy of SatelliteGeibor@aol.com): Pink Kyonshi: Sosekushi Green Kyonshi: Zanshi Fat Kyonshi: Kimenshi Dwarf Kyonshi: WeeKyonshi (literally) Tall Kyonshi: Ryukyoshi Graveyard Ghost: Shanshi --Know any GameGenie codes? Nope. Go to the Codes and Secrets page of this game and you might find something to help, you cheater! --- ## ---> WALKTHROUGH In case you don't have a manual, here is the juicy bit of the story: You are Kenchi, China's savior from the Kyonshi, the eastern counter- part of the vampire. Someone had let them loose on seven towns, with the eighth one being totally controlled by that being. Of course, you as the hero walks in the scene and puts a stop to the onslaught. However, it won't be easy: the Kyonshi are undead, has little sense of pain and their conscience all gone. Which makes them more fun to battle, isn't it? +---------------------------------+ |Town 1 - A taste of Kyonshi power| +---------------------------------+ Enter the temple. This holy building looks always the same in every town you come to; it also functions the same thing; as a rest point for your character. That's why you should also invade this first. Oh, the controls: left/right: moves you in the direction you press. up: jump (not so high right now, so don't use it much) down: crouch/duck B button: punch A button: kick Select: when outside, changes message display speed Start: pauses the game Kill the introductory kyonshi inside, then the temple could now be used as an inn. The ASCII art below shows the lay-out of the town: T S O S S O H S O I I B |_||_||_||_||_||_||_||_||_||_||_||_| Legend: T-Temple H-Training Hall I-Item inside S-Scroll inside O-Orb inside B-Boss inside Enter houses and fight kyonshies to have scrolls and some items. Pink kyonshies are weaker than green ones, but later in the game, most of the time you will fight green ones so you have to practice on them, too. Heal at the temple if you are hurt. Every time you get enough scrolls, head to the Training Hall and learn a new ability, in the order 2 Kick, Wolf Move, 2 Thrust, then High Jump. Afterwards, enter the Orb buildings, defeat the more difficult kyonshies there and get the Orb they leave. After you have three orbs, enter the boss cave and fight... AFTERIMAGE KYONSHI (Japanese name: Genyoshi) Difficulty: */ * --I just made up the name, sorry. This boss is small but jumps ultra high. Also it takes up many hits to bring down so don't you dare stand in its path to kick/punch. You could a) wait until it reaches the high point in its jump, then as it falls down, attack it, or b) my favorite way. Use the talisman to freeze it then kick it from behind. Rinse and repeat. Your reward? Being congratulated by your assistant and access to the next town =). Don't forget to copy the password. +-------------------------+ |Town 2 - The baby Kyonshi| +-------------------------+ I H S S O O T I O \\* I B |_||_||_||_||_||_||_||_||_||_||_||_| The \\* contains Conshi, the baby Kyonshi. You can have him in your party by letting your assistant be snatched by the light in the graveyard, defeating her (she's very strong, but not too hard), then afterwards you will get the bell. You may want to try it to know how it feels to be a Kyonshi; otherwise do the same modus operandi in Town 1 (learn the Turn Kick as soon as you can), and at the end is another cave... KNIFE KYONSHI (Japanese name: Raunshi) Difficulty: /*** --As soon as you enter, stop for the boss is throwing knives at you. After enough knives, he pauses, that's your signal to get near and whack him. If he hits you with his knife, it does a lot of damage so be careful. You may either use the Tonten to knock him out fast when he is about to use the knives, or the Talisman to freeze him then hit him from behind so he has to turn around to hit you with knives. And if you're wondering whether you can beat this boss using Conshi, well, YOU CAN. It's quite difficult however, and I'm not 100% sure if there are rewards somewhere, but it's quite fun watching him knock out that big guy =). +----------------------------------+ |Town 3 - Woo! A bit of difficulty!| +----------------------------------+ S I I S S T H O S O O B |_||_||_||_||_||_||_||_||_||_||_||_| Save the temple, get some scrolls, learn at the training hall, and then get the three orbs. Simple as that. LIGHTNING KYONSHI Difficulty: */*** --From here, the manual doesn't say the Japanese names, so you have to deal with the translation above. Sorry. --No, he doesn't throw balls of lightning at you. It's that the back- ground flashes every now and then. The boss itself is like any other tall Kyonshi, except he's much faster than the others, much stronger, and much more HP. +-------------------------------------+ |Town 4 - Hey! Be careful! WATCH OUT!!| +-------------------------------------+ H S S O S I O I T O I B |_||_||_||_||_||_||_||_||_||_||_||_| Don't bother learning the Wind Kick; it's pretty much useless. The other move, Mirage Thrust, is a must-have. You can easily kill tall bosses without even getting stratched. I know it's cheap, but an army of kyonshies vs. one phantom fighter isn't fair, too. TACKLING KYONSHI Difficulty: * / * --This is the part where I always got stuck when I was a kid. This boss attacks you at a speed in which you could rarely, if ever, react in time. If you don't hit him with anything in your arsenal, he WILL tackle you. Talisman does not work on him. The Tonten and the Sacred Sword does work on preventing him to attack, but does little to no damage. Try to put him into a corner then attack him whenever he goes floats back up so the chances of him tackling you will be minimized. +--------------------------------------------+ |Town 5 - Who shakes in terror? You, or them?| +--------------------------------------------+ I \\* S I S I T I O O O B |_||_||_||_||_||_||_||_||_||_||_||_| As you notice, there is no training hall. That means it is at the other town, so collect scrolls till you reach the maximum of 99. EARTHQUAKE KYONSHI Difficulty: * */*** --Why the \"earthquake\"? Because when it lands on the ground, it actually shakes it so much that Kenchi falls over. Also, when you're hit by the claws, you are sent to the other side of the room. The good thing? You could use the Mirage Thrust to post a talisman on him, or just plain punch his knees till his life meter runs out. +-------------------------------------------------------------+ |Town 6 - THIS, has gotta be, the weirdest flying kick ever...| +-------------------------------------------------------------+ H T O O O B |_||_||_||_||_||_| The moment you enter, enroll at the training hall and avail of the powerful Jump Kick. It's a shame you would learn it this late in the game. You may choose to save the temple, or if you feel really almighty, just leave it there and snatch the orbs one by one until you can enter the boss lair... AXE KYONSHI Difficulty: * */*** --He does not throw axes at you; it's his melee weapon. He just holds it out in front of him, swinging it once his near enough. Don't punch him; use your feet both for kicking and dancing around him. +-----------------------------------------------------------+ |Town 7 - What?! I thought I could use magic in this game...| +-----------------------------------------------------------+ O O S S S T H S S S O B |_||_||_||_||_||_||_||_||_||_||_||_| No items. Save the last temple, collect scrolls this one last time, then learn 4 Thrust at the last training hall. After three orbs, fight the last boss... FIREBALL KYONSHI Difficulty: */*** --Obviously, he throws fireballs. Very easy to dodge; just jump high then drop behind him. If you learned every skill, you could do it the easy way (crouch then punch) or the exciting way (stand in front of him and flaunt your amazing abilities). Just don't get burned by fireballs, 'cause it HURTS. +------------------------------+ |Town 8 - The stupid final boss| +------------------------------+ O O O FB |_||_||\\_|---|\\_\\_| The --- represents a bridge, and FB is the FINAL BOSS. Are you ready? There are no temples in here, so once you enter a building, you have to make it till the end. There's no turning back. I just hope you really mastered those moves... OBO Difficulty: */*** --First time I said she was easy, I forgot you have to fight an army of assorted Kyonshies to get to her :). --You're fighting, for the second time (the first was a ghost, remember?), a non-kyonshi. Besides teleportation and magic fireballs, she does nothing but stand there, waiting for you to hit her. Unless you are cursed with kyonshi-slow reflexes, she is very disappointing. Sure wish she was a lot more powerful. You may watch the credits roll, but you won't get a new game. Now it's all over. Kyonshies will never bug you again =). --- ## ---> CREDITS My father, for buying me a cartidge a looong time ago (back in 1989); FCI, for making the game; Nintendo, for a really good console that'll last for ages; CJayC, for hosting the guide on his site, GameFAQs; SatelliteGeibor@aol.com, for the Kyonshi names; and you, for playing the game. --- Contributor Recognition: http://www.gamefaqs.com/features/recognition/37592.html --- +------------------------------------------------------------------+ Phantom Fighter FAQ |>>>>>>>>>>>>>>>>>>Copyright 2003 Aaron Madrinan<<<<<<<<<<<<<<<<<<<| +------------------------------------------------------------------+ --- Phantom Fighter FAQ by The Deacon e-mail:mmacinni@cs.oberlin.edu --- Before we start, even though they are probably out of business, the game \"Phantom Fighter\" for the NES is trademark copyright etc. FCI and whoever else made it. Feel free to toss this FAQ to the winds, just don't be a lamer and put your name on it; if you have anything to add, e-mail me and I'll insert your info (as well as credit you with the addendum). ## Contents I.Why is this in FAQ format? I. Why is this in FAQ format? I know the term \"FAQ\" gets bandied about a lot, but since I'm something of a wordsmith I feel that we should nail down exactly what a \"FAQ\" is. You are reading a \"FAQ\", a compilation of \"Frequently Asked Questions\". Christ, if you have net access you should know by now what a FAQ is. Anyhow, it's also a FAQ because walkthroughs are really lame. The way I see it, any kind of aid on a game signals one's inability to rise to the challenge, however in the greater scope things I figure we can call getting level passwords the \"little white cheat\", reading a FAQ a medium or venial cheat, and working the keyboard/mouse with one hand while holding the walkthrough in the other and looking at the screen only to perform the walkthrough's dictated actions is a \"mortal\" cheat. And besides, walkthroughs are really only the kind of thing I could see a reasonable person using in a game with some kind of plot to it. I know a fellow who isn't very good at video games at all, and with as little patience as he has, when he gets a game he gets the walkthrough and goes through the game just to see the plot. However, Phantom Fighter (how's bout we shorten that to \"PF\") has _no_ plot. Really. You'll see. II. What's the plot? I just told you. There isn't one. (see above). However, if you want a basic rundown of the pseudo-plot, well then here you are: You are Kenchi, martial artist and aspiring Phantom Fighter. With your trusty sidekick you travel from town to town seeking fortune and fame as a wandering hero, driving out evil hopping vampires and rescuing innocent townspeople using only your wits, a few sacred items, and your superior kung-fu. That's about the size of it. III. What's the point of the game? Well, as with all games, it's to have fun...:). Besides that, there doesn't really seem to be much of a point...if you were to say that the \"point\" of the game is to be defined as what is required to beat a game, then I suppose there is a point. That definition makes the point of this game thus: You go from town to town (there are 8 in all), driving out vampires with your fists (and feet) of fury. Along the way, you pick up sacred items and ancient scrolls (more on them later) and improve your kung-fu until the final confrontation with the evil witch Obe. (\"obay\") IV. Anything I should know before I start the game? Since I didn't have the manual I'll take the time to explain the basic game mechanics for you. A: Kick. (or, if you prefer, \"Boot to the Head\") B: Punch. Start: Pause the game. Select: Does nothing as far as I know. V. What's the basic pattern of the game? Basically, you'll come to a town with your sidekick in tow. The mayor (I suppose) comes out and kowtows to you, weeping with gladness; for they called the thunder and now they got it. When you enter town, you'll be in a far-off side-scroll view with Kenchi and his sidekick walking around near buildings. When in front of a door, push A or B to see a brief message telling you what's going on inside. (I'll explain them later.) To enter a house, stop in front of the door and press Up. When you enter the house, your energy will be displayed on the left side of the screen, and any enemies in the room will have theirs displayed on the right. As you will soon see, a purple bar means you have full health, and as you take hits the little boxes turn pink until you get wiped out (and your sidekick speaks ill of the dead). In this screen, pressing up makes you jump, down makes you crouch. You cannot attack while crouching. (A BIG PISSER IMHO) Obviously, when you see a vampire come hopping up at you, kick and punch it until it falls over and bursts into flame, then continue to the right. Events vary, but after getting to the end of the house you will be presented with either scrolls or a sacred item, sometimes information (other times just blather). Some houses contain lots of enemies, make your way to the end, defeat a moderately tough enemy, and collect a jewel that instantly refills your health. You'll see it sit one of the empty stands in the upper right corner of the screen. Collect three of these things and you will have the magical power necessary to break a sealed portal. When you have three gems, you can enter the building that is always all the way at the far right end of town. Inside is a town boss, the source of the vampire scourge (for this area, anyhow). Teach him a lesson (\"Don't think. Feeeeeeelll!\") and you'll enjoy being toadied to by your sidekick. Then the happy mayor will give you a password and you're on your way to the next town. VI. What the hell ARE these things I'm fighting? Those, gamer, are hopping vampires -- China's version of the ubiquitous super- natural parasitic entity that has frightened many a superstitious person for hundreds of years. I believe the game's term for them -- \"Kyonshi\" -- is Mandarin Chinese for \"hopping vampire\". However, it could be a bastardized Chinese put into a game for mainly Japanese players. I think in Cantonese the term is \"Jiangshi\". I'll refer to them throughout this document as \"Kyonshi\", simply because that's what the game calls them. What makes these guys different from their Romanian \"cousins\"? Well, their origins are slightly different. Kyonshi are \"undead\", as in those who have died, but somehow have had their souls trapped in their body instead of escaping to the Nine Heavens (or Hells) after death. This could be for any number of reasons; bad chi flow through the house, suicide, lots of angst, etc. Anyhow, because they have died, their bodies aren't alive, they're just \"undead\". This means that they're not as limber as they once were, hence the hopping; it's the only way they can get around. What motivates them? Not much. Unlike the Western vampire, a Kyonshi has little or no conscious thought. Its only thought is to attack living things and eat their bodies. Since they don't consume their victims whole their victims become vampires as well. Kyonshi don't speak, don't respond intelligently to problems. This is a noticable advantage that the phantom fighter has over his unnatural opponents. A nimble phantom fighter should dance about to keep his assailant's long fingernails (so sharp they're almost claws) at bay. For those interested in Kyonshi I'll provide a bibliography at the end of this FAQ -- they're pretty cool. It also helps to know something about them for later (see below). VII. Do the towns follow a basic pattern? Yes. For the most part they are a bunch of houses, a temple, a \"boss shack\", and a martial arts studio. You can always recognize the temple and the studio, they're the same in every town. The temple is a large building with pillars in the front and lots of pillars and Buddha statues inside. The studio is a long, one-level wooden building. There will be three places where you can get gems as well, and the rest of the little shacks round out the town with places to kill Kyonshi and get ancient scrolls. This seems as good a time as any to explain what the little messages you get when you press A or B in front of a house mean. \"Kyonshies are here\": Ahem. \"Kyonshies are not here\":Ahem. Ahem. Okay, besides the obvious, these mean there is just a basic bunch of Kyonshies in the house (or the house is empty). Fight through all of these and you'll get some ancient scrolls. Empty houses seem to have no point; you walk all the way to the end and some cowering peasant thanks you or says something worthy of the term \"non sequitir.\" \"This is a temple.\": What the Christ. It's a temple. \"There's danger in the air\": This means that inside is either a sacred item or one of the three gems you need to bust them ghosts. \"What, nothing's happening?\":or something like that. Basically one of those houses with goodies or gems in them that you've cleared out already. \"Open the sealenter\": Obviously an error in the programming, this should read something like \"Open the seal to enter\". This is the \"boss shack\" that you need the gems to get inside. \"Enter with courage\":When you have the gems, you can open the seal to the \"boss shack\" and instruct the Kyonshi on the finer points of kung fu. (\"I hope you were paying attention.\") VIII. From the looks of things, I'm not a very good phantom fighter. I just got my butt kicked. Any way to improve on this sad situation? Fortunately, yes. At the beginning it may seem like you'll have a lame kick and punch for the entire game, but fear not. You can improve to such powerful kung-fu that you'll easily steamroll enemies. So, how do you go about improving your skills? Easy. First, you have to do a little struggling; you need to earn ancient scrolls. Just pick a small house and go in. When you get some scrolls go to the studio. When you enter, a fat guard accosts you, and asks you if you understand that you won't get kung-fu lessons for free (This is basically to spare you from the annoying un-skippable dialog that follows if you don't want to enter after all). When you say that you understand, he asks you a question, to test your knowledge. A. What the hell is this? Copy protection? I have no idea why this in the game. I can only guess that at the time this was made, it made sense, or really was to protect from people pirating the game. I don't have the manual, so I don't know. Don't sweat it too much, the questions are pretty easy, often it's obvious what the correct answer is, and if you fail you can always try again. B. Fuck that. Just give me the questions and answers. Ok, I guess quiz games aren't your thing. Here are some of the questions that the fat guy asks, and their answers: (These are from memory, more to come) \"Name an FCI video game.\" I think all of these are FCI games, pick \"WCW\" just to be sure. \"Why do Kyonshies only come out at night?\" They hate the sun, obviously. \"How do you capture a Kyonshie?\" Although you never see it in the game, you use an Urn. \"What is a Kyonshies least favorite food?\" I don't know where this hell this came from, by process of elimination it's ice cream. When you answer correctly the fat guy lets you pass and you can enter the Master's chambers, where they gather for the feast. And though they stab it with their steely knives, they just can't kill the beast. Sorry. If you got that reference you'll know the special sort of anguish I feel now at selling my copy of that album. Anyhow, the master will train you in kung fu in exchange for ancient scrolls. Why, I don't know, but my motto is do what works. So, cough up scrolls and train in the martial arts. The different moves you can select are listed with their scroll cost to the left of them. X. What are the different kung-fu moves, how much do they cost, and what do they do? Good question, grasshopper. There are three basic move categories: \"moves\" (\"mv\"):your movement \"thrust\" :punching moves \"kick\" :ahem, kicking moves. Here's a breakdown by category: MOVES: Wolf Move (\"wolf mv.\") Cost: 2 scrolls What it does: Makes you move faster. Useful. Tiger Move (\"tiger mv.\") Cost: 6 scrolls What it does: see above. Mirage Move (\"mirage mv.\") Cost: 6 scrolls What it does: see above. Dragon Move (\"dragon mv.\") Cost: 18 scrolls What it does: see above. Mirage Walk (\"mirage wk.\") Cost: x scrolls What it does: Allows you to walk while crouching. Mirage Thrust (\"mirage th.\") Cost: 50 scrolls What it does: Allows you to punch while crouching. A terrific move. High Jump Cost: x scrolls What it does: you jump higher. Absolutely necessary for airborne vampires. Windmill Jump (\"wind jump\") Cost: x scrolls What it does: you jump still higher, and you flip in the air as you do. Allows you full range with airborne enemies and looks really cool. THRUSTS: 2 Thrust Cost: 2 scrolls What it does: Gives you two lightning-fast punches instead of one. 3 Thrust Cost: x scrolls What it does: Basically same as above, except now it's a 1-2-3 punch. 4 Thrust Cost: 90 scrolls What it does: Now, you punch both high and low, twice. The most powerful move in the game, believe it or not. KICKS: 2 Kick Cost: 6 scrolls What it does: the 1-2 kick. One for both ears. :) Side kick Cost: 18 scrolls What it does: you lean to the side and use your hips to put a little torque action into your kick. Good damage, another move you'll use right up to the end. Windmill Kick (\"wind kick\") Cost: 50 scrolls What it does: When you jump straight up in the air and kick, you'll spin around in a flurry of feet. Really only useful for airborne enemies, and doesn't always come off clean. Jump Kick Cost: 80 scrolls What it does: Run at your enemy and kick and you'll leap at them with both feet forward. (\"Mind your Manners!!!\") Great as an opening move, as well as for enemies who like to jump a lot. XI. What are some good fighting tactics? Stick and move. If you stand in one spot you'll get creamed, as more often than not enemies will soak one or two kicks and swipe you with them claws. This is most important at the beginning of the game, when you can't do anything but punch and kick, and move around a little bit. The main method of attack Kyonshies use is to take little hops at you, then when they get close enough to a big Kyonshie leap into your face. What you want to try to do is to place yourself at a point where the Kyonshie will be in midleap when it comes into kick range. Then, when it leaps, kick it, and it'll fall over. It won't die, but knocking it down gives you breathing room. You'll take a lot of hits from Kyonshies just steadily leaping towards you. That's their other fighting tactic...they just keep coming, soaking up the kicks and punches until they can cut you. If you can kick them enough times, however, they'll fall over. Different kyonshies take more or less kicks/punches in a row to be knocked down, size is one factor. Later, when you learn more kung fu, it will be easier to defeat enemies, although the enemies will get progressively harder as you go from town to town. One good move that will help you out a lot is the \"Wind Jump\". This allows you to leap high into the air (doing a cool kung-fu flip) above the reach of the dreaded hoppping demons. When enemies come at you, kick them once or twice, and if they don't fall down hop over them just before they get to you. A little practice helps, but soon you will be able to drop down right on the other side of them and kick them in the back of the head while they are turning around. Kyonshies are very stupid. It helps a lot to remember this, as the next trick shows. See, Kyonshies (as you will know if you read the above section on them) are in the throes of rigor mortis, their bodies in an arrested state of decay. So they can only see straight forward, right? So if you crouch down, they will turn back and forth, their tiny minds trying to figure out how you pulled your disappearing act. This will not be very useful in the beginning, as you can't walk while crouching. DON'T try this just before a Kyonshie gets to you -- their turning back and forth motion will still be able to hit you, as you are not completely under their arm level (unless you are fighting a really tall kyonshie, see below). This move becomes more useful when you learn the Mirage Walk, as you'll be able to shuffle about under their vision, just don't attempt to get too close to them as they can often still hit you. Later, when you learn Mirage Thrust, this becomes a winning tactic, allowing you to stand under tall enemies and bosses and punch them in their unfeeling vampire gonads mercilessly. (WARNING: This is kind of a cheapo move, as you really can kill tall kyonshi bosses with this tactic. It certainly doesn't _look_ very exciting.) XII. Alright. I'm in town. What do I do first? Wander through town until you find the temple, then try to fight your way through it, shouldn't be to difficult. In the first town, it's one of the first buildings, later on it will be farther into town, or sometimes nonexistent. Anyhow, it's important to start here, so that you can fill up your health, you'll need to often and there's only one other way to do it in the game, and that involves more fighting. Once you've done that, go to the kung-fu studio and see how many scrolls you'll need to learn the various kung-fu moves. The dialog trees can be annoying, but get used to them because they are all through the game. :P Anyhow, leave the studio with the moves you want to learn in mind, and find a house with Kyonshies in it. Then enter and work your way to get the scrolls. Surprisingly (or perhaps not, depending on how well versed you are in \"game logic\"), you can enter the same house again and again, continuously obtaining scrolls. You can get up to 99, but don't bother trying that here unless you are sick in bed with chicken pox or something -- you'll have better oppportunities later on when you recieve lots more scrolls for rescuing townspeople. Learn all the moves you can, when the master tells you you've tapped him out, you can move on. A. Just to make it easier on myself, what moves should I learn first? Since your kung-fu is puny, learn attack moves first. Specifically, get 2 Kick and 2 Thrust. Throughout the game, when you have an opportunity to learn an attack move, learn it, and learn the other moves later. While the \"Move\" moves may seem pointless, they are actually very useful, since being able to move quickly will allow you to dance in an out of a kyonshie's range without getting creamed every time you try to hit them. It's also highly important to learn the jump moves, since they allow you to attack airborne enemies, of which there are 3 in the game, all of them important. While I suppose you could beat the game without all of the moves, I wouldn't recommend it, simply because it wouldn't be as fun, as learning new moves is one of the high points of the game. After you've learned all the moves you care to, head for the houses with the gems in them, collect the gems, and head for the boss shack. Don't forget to write down your password! XIII. Hey, these Kyonshies aren't all alike! Quite right. There are several different types of Kyonshies. Basically, there are four basic types of Kyonshi, differentiated by body type: the little one, the medium one, the tall one, and the big fat one. Within these different morphical categories there's lots of room for variation in the different colors they wear. You'll no doubt become quite familiar with the types as you play the game, and certain ones will become infamous. (\"ARgh! Not more green ones!\") XIV. Okay, so there's different types. What are they like? Without further ado: little: This little bastard is fast and hard to hit. He'll be a toughie when you're duking it out in the first town and don't have awesome kung-fu. His basic weakness is that he can't jump that far, and he's really short, making him easy to hop over. Fortunately he's still just tall enough to kick in the face. medium: Your basic vanilla Kyonshi. Some types jump a lot, some are fast, etc. tall: The tall guy moves ponderously slow, and thanks to his height is hard to jump over. He also jumps very high while attacking as well. Naturally enough this is a big mistake on his part; he jumps so high that you can crouch and let him go over you, then \"boot to the head\" while he's turning around. Later on when you learn the Mirage Thrust the tall guy will be at your mercy as your firsts pound mercilessly into his undead loins, and his turning back and forth schtick won't be able to nail you. fat: This guy is tougher than he looks. He's very strong, such that even though he moves slowly, he packs a real wallop. Also, he doesn't jump that high, but is still tall enough that jumping over him is a bit of a sticky wicket. Your big advantage over him is speed. While he can soak up hits and really wummox you when he gets a hit in, if you play it right you can kick once or twice then get the hell out of the way when the Kyonshi express comes through. XV. I won some kind of item, what does it do? Aha, one of the coveted mystical items that you can use in the game. Don't confuse these with scrolls -- ancient scrolls sound cool but don't do jack except buy you kung-fu. However, the items are quite nice. First, here's how to use an item once you've got it. Whenever you obtain an item, you will see it appear at the top of the screen, in a long box to the left of the stands that hold the three gems you'll need to get out of town. However, make no mistake, you are NOT carrying that item. Your faithful sidekick (who has no name that I could tell) is carrying them. Thus, whenever you enter a building, and you want to use an item, turn back as if to leave. Your boy will pop his head in and ask if you want to split, use an item, or ignore him and get back to butt-kicking. Well, ask for an item, and he'll give you a list: strangely enough, he lists all the items you can get, even if you don't have them yet. Pick one, and it will be outlined by a red square at the top of the screen. This means you are now carrying the item, and will use it instead of punching when you push A. Now, here's all the items, and what they do. Talisman: Freezes enemies in their tracks sometimes. Tonten: Knocks baddies onto their cans with a flash of light. Sword: Knocks guys down, you can also slice 'em with it. Bell: Controls the demon-boy. A note of warning: your items will also take hits for you if you get hit. However, this will also break the item, making it useless. Hey, it's an antique. Keep this in mind when fighting with an item. It's a good idea to grab them, they're nice for keeping enemies at bay, and the Tonten is pretty damn invaluable for beating the game. XVI. I bought one of those Game Genies. Any codes for this game? CODE KEY IN . . . EFFECT . . . 1 VTVKEGSA + KAVKOGNA Start with Sword 2 VTVKEGSA + SAVKOGNA Start with Bell 3 VTVKEGSA + UAVKOGNA Start with Tonten 4 VTVKEGSA + XAVKOGNA Start with Talisman 5 LASKNGAA + VAVKOGNA Start with 3 Scrolls 6 TASKNGAA + VAVKOGNA Start with 6 Scrolls 7 SXSZLUSE Infinite energy 8 OVSZPLSV + PESZZLAA Take less damage when attacked (author's note: This is taken verbatim from Galoob's master list of all Game Genie codes for NES games that they ever published. There may, of course, be some home- grown codes floating about. So all due credit goes to Galoob for the above blurb.) XVII. This isn't the most complete FAQ I've ever read, you know... Sorry. This is mostly from memory. If I get e-mails requesting updates I'll saunter back into the game with my rightfully earned cheat codes and research it fully for those who desire it. And besides, if you think this FAQ is incomplete, check out some of the other (admittedly) sparse fare at TSR's NES FAQ page. For right now I'll call an end to this FAQ, just because I figure this is enough to be written about any NES game that wasn't made by Square. :) I'll send my regards first and foremost to TSR, who's NES page is truly one of the best place for NES info on the 'net, not to mention skilled use of graphics and delightful content like the NES oddity page. Also, here's to the place that posted the ROM I used for this FAQ, Big Daddy's International House of ROMs, THE place to get ROMs on the 'net. In my experience, if he doesn't have it here, it doesn't exist yet (at least for NES, anyhow). I'll also send out greets to John Turk, who's NES Underground Library is probably the most ambitious of all the NES pages I've seen. And finally I'll send shots out to all the other NES pages, all those who've stayed cool despite NESticle's appearance, and to all the fellows on IRC channels like #emuroms, #emulator, and #emu who are out there circulating coolness in less than 200k. :) Also a shot out to my friends on #1980Warez, who have nothing to do with NES ROMs, but are cool anyway. Jibes go to all those who feel the need to charge excessive amounts of money for their emulators. Say what you will about effort and reward, guys, but in the end you're creating something used to do something that's, shall we say, a trifle off-color in the eyes of the law. Jibes also go to those who run IRC channels so huge that no chatting goes on in them, and channels that have nothing but DCC bots in them. Thanks for helping turn IRC into a wasteland, guys. So much have I written for gamers, now give me a drink! The Deacon Restore Page Training Hall Quiz Questions and Answers +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^+ < ////// // // /// //// // ////// ///// ////////// > < // // // // // // // // // // // // // // // > < ///// ////// /////// // //// // // // // // // > < // // // // // // /// // // // // // // > < // // // // // // // // ///// // // // > < > < ////// // ///// // // ////// ////// ////// > < // // // // // // // // // > < //// // // /// ////// // //// //// > < // // // // // // // // // // > < // // ///// // // // ////// // // > +------------------------------------------------------------------+ * * * * * * Training Hall Quiz Questions and Answers * * * * * * A guide by PinKirby (Pinku_Kirby@yahoo.com) +-----------------+ ----Version 1.0---- +-----------------+ June 9th, 2004 -Began this guide -------------------- Contents: 1. Introduction 2. The Training Hall 3. The Questions and Answers 4. Possible Messages 5. Thanks 6. Legal Info --------------------- * * * * * * 1. Introduction * * * * * * Phantom Fighter is a game that is not very well-known, but is still, in my opinion, a great game. I've decided to make a guide on the questions the guard of the Training Hall asks you. Most questions have one correct answer, but some questions are asked in which ALL of the multiple choices are correct answers. Here's how to use this guide, using a sample question NOT asked in the game: ***QUESTION: What game does this guide cover? 1. Super Mario Brothers (X) 2. Phantom Fighter (O) 3. This game has no name(X) (X) - Means an incorrect answer, in which the guard throws you out. (O) - Means a correct answer, in which the guard lets you pass. --------------------------------------------------------------- * * * * * * 2. The Training Hall * * * * * * After defeating the monsters, the sharp-nailed Chinese phantoms which are known as \"Kyonshies\", in houses and buildings, you can get one or more scrolls. You use these scrolls to learn skills in the Training Hall, but you can't see the master for free, as his guard explains to you. To prove you are worthy of the master's time and attention, you must correctly answer a question with one of three multiple choices. While many of the guard's questions pertain to your foes the Kyonshies, some of the questions involve trivia. If you get a question wrong, DON'T FRET! You can re-enter and try again with another question. The guard tells you, Kenchi: \"You can't see the master for free. Do you understand?\" If you answer no, he'll laugh at your cowardice and kick you out. Answer yes, and it's quiz time! --------------------------------------------------------------- * * * * * * 3. The Questions and Answers * * * * * * QUESTION 1: How many stars are there in the American flag? 1. 50 (O) 2. 28 (X) 3. 5 (X) --------------------- QUESTION 2: What country do samurais come from? 1. India (X) 2. Japan (O) 3. Spain (X) --------------------- QUESTION 3: What is the best method to make sure Kyonshies never revive again? 1. Big Stakes (X) 2. Seal in Rock (X) 3. Fry in Oil (O) --------------------- QUESTION 4: What is the teaching taught by Confucius called? 1. Psychiatry (X) 2. Confucianism (O) 3. Physiognomy (X) --------------------- QUESTION 5: Who built the Great Wall of China? 1. The Emperor (O) 2. Plasterer (O) 3. Farmer (O) --------------------- QUESTION 6: What is Kyonshies' most powerful weapon? 1. Sharp Claws (O) 2. Sharp Eyes (X) 3. Swift Moves (X) --------------------- QUESTION 7: What is the horrible skill called when your head is banged against your opponent's head? 1. Head Strike (O) 2. Head batting (O) 3. Head to Head (O) --------------------- QUESTION 8: What's the name of George Bush's dog? (This is George Bush Sr.!) 1. Millie (O) 2. Martha (X) 3. Pooch (X) --------------------- QUESTION 9: Who created Ultima? 1. Lord British (O) 2. Prince Chuck (X) 3. Mr. Exodus (X) --------------------- QUESTION 10: What is the best thing to use to capture Kyonshies? 1. Coffin (X) 2. Urn (O) 3. Jewelry Box (X) --------------------- QUESTION 11: What is the famous Chinese Newspaper? 1. School Paper (X) 2. Wall Poster (O) 3. Daily China (X) --------------------- QUESTION 11: What is the food Kyonshies hate? 1. Liver (X) 2. Ice Cream (O) 3. Frog Eyes (X) --------------------- QUESTION 12: Something is used to beat Kyonshies. What is it? 1. Bird's Blood (O) 2. Lizard Scale (X) 3. Vulture Nail (X) --------------------- QUESTION 13: Why do Kyonshies only come out at night? 1. Play late (X) 2. Shy (X) 3. Hate the Sun (O) --------------------- QUESTION 14: Which emperor is often called the Last Emperor? 1. Sagi (X) 2. Higi (X) 3. Fugi (O) --------------------- QUESTION 15: What kind of place do Kyoshies usually live in? 1. Wet place (O) 2. New Jersey (O) 3. Beverly Hill (O) --------------------- QUESTION 16: What is the skill called when you hold your opponent's arms from the back and throw him backward? 1. Dragon Suplex (O) 2. Front Suplex (X) 3. Side Suplex (X) --------------------- QUESTION 17: Name an FCI Video Game. 1. Hydlide (O) 2. WCW (O) 3. Ultima (O) --------------------- QUESTION 18: What is a Chinese martial art usually called? 1. Kung Fu (O) 2. Martial Art (X) 3. Phantom Kick (X) --------------------- QUESTION 19: What is another word for one? 1. Uno (O) 2. Ichi (O) 3. Un (O) --------------------- --------------------------------------------------------------- * * * * * * 4. Possible Messages * * * * * * ----- CORRECT MESSAGES: - That's good. The master is waiting for you. Go and see him. If you answer \"New Jersey\" or \"Beverly Hill\" to the question about where Kyonshies usually live in: - That's right. Now as you know, that is the hangout of Kyonshi. Go and enter. (I'm assuming the guard pities you so much that he just accepts it as a correct answer. :P ) ----- INCORRECT MESSAGES: - Why are you trying to pretend you don't know the answer? - You, Kenchi, not know a thing like this? Worth a laugh! Ha ha ha! Now, get out! Get out! - That was a pity. Train yourself and come again. - How little you know I'm stunned. Think it over and come back again. - Kenchi, you are really an idiot! Go home, try to learn more, and come back when you are wiser. If you answer \"5\" as to how many stars are on the US flag: - Dunce! --------------------------------------------------------------- * * * * * * 5. Thanks * * * * * * - To my sister, Mom and Grandpa - To FCI and Pony Canyon for making this game - To YOU for taking the time to read this guide! * * * * * * 6. Legal Stuff * * * * * * This guide is copyrighted (c) by PinKirby. You may copy SOME of this guide, but you MUST give me, PinKirby, credit. If you want to put this guide up on your website, email me and I will see about it. If I find that this guide is on a website NOT listed below, then I will have it removed. SITES THAT MAY HAVE THIS GUIDE UP: www.GameFaqs.com Phatom Fighter is (c) 1989 by FCI and Pony Canyon. The Nintendo Entertainment System and Nintendo of America are registered trademarks of Nintendo. All rights reserved. This Phantom Fighter Training Hall Quiz Guide is (c) by PinKirby Restore Page","title":"nes"},{"location":"nes/#nes-faq","text":"","title":"nes faq"},{"location":"nes/#phantom-fighter","text":"==================================================================== Phantom Fighter Complete FAQ/Walkthrough September 05, 2003 Version 2.00 author: Aaron Madrinan (snoocete) e-mail: snoocete@yahoo.com subject: \"RE: Phantom Fighter FAQ/Walkthrough\" ==================================================================== --- ## ---> INTRODUCTION The gaming world is changing very rapidly. Games are released on breakneck speed that we need more than a lifetime to play even just the good ones. The result? The other not-so-famous games are forgotten and put aside by future gamers. In other words, they become underrated. Such is what happened to a game called Phantom Fighter. I've played it since I was a kid. Though not as extremely fun as games like Kirby or Pokemon, it very much dwells on the concept in which Contra flourished: you can never let the game beat you. Supposedly, it would be as famous as that. But, heck there wasn't even a walkthrough on GameFAQs, only some codes that lets you skip to the last town and a very bad review. Sigh. You want my opinion? Here is what I can say: don't let the big names fool you. A lot of gems are hidden in the dump of really bad games, I admit, but they only need a little spotlight to shine. Here is my share of that spotlight =). --- ## ---> DISCLAIMER You are free to save this guide onto your hard drive and/or print for your personal viewing pleasure. You may distibute it to other people as long as you don't claim it as yours, okay? Since this game doesn't have that much coverage, you may post this at your site in its full, unedited version even without my permission as long as its credits point to me. Deal! --- ## ---> CONTENTS 1. Updates --- 2. FAQ --- 3. Walkthrough --- 4. Credits --- This is FAQ #2, hope I got a bit better. --- ## ---> UPDATES -2.00- -21Kb- Look above (^\\_^). It's a whole number change! A kind soul mailed to me the names of the Kyonshies. Yay! Added peace and order (where there was chaos). Formatting change. E-mail change. Lastly, look at the disclaimer. -1.00- -19Kb- This FAQ is complete, however the Japanese names of the Kyonshies are missing. Not that anybody would care, but it still important. If you have an instruction manual (because I kinda lost it), kindly e-mail me and let me know, ok? --- ## ---> FAQ --Who are you? Just a humble freelance (a bit newbie-ish, but wants to learn a lot) FAQ writer who writes for his favorite games whenever he has the time. --What are these \"Kyonshi\" that I'm fighting? If you have ever watched the anime series \"Shaman King\", or knew a bit about Chinese folkore, you have the idea. These are what we call the undead; they are supposed to be immobile, rotting bodies. They still are, but they are being controlled by some mysterious power. You know about rigor mortis? It also applies to these phantoms. That explains why they could only move their feet and knees to hop and turn around, and their elbows to puncture your face with their dirty, sharp nails. --How do I fight these enemies? You have to move around. A lot. Whenever you stay on one spot, the Kyonshi will follow you with small hops, then when it gets in range, it suddenly does a big dive toward you. Unless you hit the Kyonshi in a critical point or you kill it in transit, you cannot stop that from homing to you. So you got to move. That's your ace against these creatures. They have lockjaw and every known bone and muscle disease, while you could just move out of their way. For example, the moment they land on the ground, give them a kick or two on their sleeves then they will be knocked down. When they wake up and hops around again, repeat. Soon they will burst in flames, and the door that lets you continue through the building opens. --What does that man who follows me around do? That's your assistant. Whenever you are inside a building, if you return to the exit, you talk to him. He asks if you want to have an item, stay at the building, or just leave the place. --And those items are...? There are a total of four (4) items in the game. You can find them from the households you rescue from kyonshies. They replace the use of the punch attack, but it's worth it. 1. Tonten - the kyonshi you show this with is knocked back to the ground because a) there is holy white light coming from it, or b) he saw his face, squirming with maggots (hey, thanks mate for the mirror, 'bout time I fix my hair... what the he... _faints_). DON'T use this when you're about to get hit; it also acts as your armor, and because it's a mirror and was hit by diamond hard claws, you know what happens. 2. Sacred Sword - even a swordless Link (of Legend of Zelda) would not use this nicely-named weapon (he might even throw it back at you so don't even try). It does little damage, and it also gets broken easily as if it is made by glass. The only edge it has is that it has a rather long range for a physical attack, you could corner a kyonshi with it, but you would like to beat them up yourself, wouldn't you? 3. Talisman - freezes the enemy in place until it's effect is gone or they got hit. You have to make contact with the kyonshi you want to victimize it with, so there is a chance you get hit and the item breaks. Useful on the tall kyonshies, you could sneak under them, paste the talisman on their belly, then kick them in the butt. 4. Bell - If you use this in the middle of a heated battle, it does nothing except probably makes a little chime that is pleasant to hear for you but instead boils the blood of the kyonshi, making him more aggressive. Actually, use it to control the kyonshi kid. --What are the scrolls for? Whenever you cleanse a building from kyonshies, you get a varying amount of scrolls. You use them at the Training Hall to learn new abilities. --The guard won't let me in because I can't answer his questions... got a list there that could help me? Please? Of course I do. I intend to make this the complete-est FAQ you'll ever see (for PF, of course). Here they are (if you noticed, the stupid comments are gone. just learned not to learn from the likes of ArchNacho and Tortilla Godzilla): -Format- Q - Question A - Answer Q: Why do Kyonshies come out only at night? A: Hate the sun. Q: What country do Samurais come from? A: Japan Q: What is another word for one? A: Any answer will do. Q: Who built the Great Wall of China? A: Any answer will do. Q: What is the food Kyonshies hate? A: Ice Cream Q: Who created Ultima? A: Lord British SC: Who is he? Q: What is Kyonshi's most powerful weapon? A: Sharp claws Q: What is the skill when you hold an opponent's arms from the back and throw him backward? A: Dragon Suplex Q: What kind of place do Kyonshies usually live in? A: Any answer will do. Q: What is a horrible skill when you head-banged against an opponent's head? A: Any answer will do. Q: Name an FCI video game. A: Any answer will do. Q: What is the famous Chinese newspaper? A: Wall poster Q: How many stars are there in the American flag? A: 50 Q: What is the best thing used to capture Kyonshies? A: Urn Q: What is a Chinese martial art usually called? A: Kung fu Q: What is the best method to make sure Kyonshies never revive again? A: Fry in oil Q: What is the teaching taught by Confucius called? A: Confucianism Q: What's the name of George Bush's dog? A: Millie Q: Who is the emperor called the Last Emperor? A: Fugi --Can I kill the guard? If you are not a super nerd, who knows NES rom-editing to change the layer which the guard resides to your characters layer and set its attributes to \"hittable\", has an HP count, and many other things, definitely no. I know it's a big disappointment. --What are these new abilities you will learn from the Training Hall? New techniques in martial arts. When you started the game, you have a weak punch and a weak kick. Learn these and you will become stronger and better to fight kyonshies. Town 1 (2scr) =2 Thrust - punch and you will hit the enemy twice =2 Kick - same with 2 Thrust but you kick instead =High Jump - self-explanatory =Wolf Move - speed up Town 2 (6scr) =3 Thrust - three fists-of-fury in one swift delivery =Turn Kick - most versatile move in the game =Wind Jump - jump and hold left/right to do a spin jump =Tiger Move - more speed up =Mirage Move - combine with Tiger Mv. for better effect Town 3 (18scr) =Mirage Walk - move while crouching =Dragon Move - whoosh! last speed upgrade Town 4 (50scr) =Wind Kick - jump then kick furiously in mid-air to form a spin kick =Mirage Thrust - punch while crouching Town 5 = Town 6 (80scr) =Jump Kick - hold left/right then kick to see this fantastic move Town 7 (90scr) =4 Thrust - most powerful move in the game Town 8 = --What are the names of these Kyonshies? What I have here are unofficial names which I made up by their looks, color, and strength: Pink - weak household form of kyonshi. Green - stronger and faster than the pink Dwarf - a real pain, their size and speed make them hard to hit. Tall - vulnerable to punch attacks, mirage thrust Fat - looks tough but slow, easy target Short-armed - has a high resistance to hits, has high attack power Their original Japanese names (courtesy of SatelliteGeibor@aol.com): Pink Kyonshi: Sosekushi Green Kyonshi: Zanshi Fat Kyonshi: Kimenshi Dwarf Kyonshi: WeeKyonshi (literally) Tall Kyonshi: Ryukyoshi Graveyard Ghost: Shanshi --Know any GameGenie codes? Nope. Go to the Codes and Secrets page of this game and you might find something to help, you cheater! --- ## ---> WALKTHROUGH In case you don't have a manual, here is the juicy bit of the story: You are Kenchi, China's savior from the Kyonshi, the eastern counter- part of the vampire. Someone had let them loose on seven towns, with the eighth one being totally controlled by that being. Of course, you as the hero walks in the scene and puts a stop to the onslaught. However, it won't be easy: the Kyonshi are undead, has little sense of pain and their conscience all gone. Which makes them more fun to battle, isn't it? +---------------------------------+ |Town 1 - A taste of Kyonshi power| +---------------------------------+ Enter the temple. This holy building looks always the same in every town you come to; it also functions the same thing; as a rest point for your character. That's why you should also invade this first. Oh, the controls: left/right: moves you in the direction you press. up: jump (not so high right now, so don't use it much) down: crouch/duck B button: punch A button: kick Select: when outside, changes message display speed Start: pauses the game Kill the introductory kyonshi inside, then the temple could now be used as an inn. The ASCII art below shows the lay-out of the town: T S O S S O H S O I I B |_||_||_||_||_||_||_||_||_||_||_||_| Legend: T-Temple H-Training Hall I-Item inside S-Scroll inside O-Orb inside B-Boss inside Enter houses and fight kyonshies to have scrolls and some items. Pink kyonshies are weaker than green ones, but later in the game, most of the time you will fight green ones so you have to practice on them, too. Heal at the temple if you are hurt. Every time you get enough scrolls, head to the Training Hall and learn a new ability, in the order 2 Kick, Wolf Move, 2 Thrust, then High Jump. Afterwards, enter the Orb buildings, defeat the more difficult kyonshies there and get the Orb they leave. After you have three orbs, enter the boss cave and fight... AFTERIMAGE KYONSHI (Japanese name: Genyoshi) Difficulty: */ * --I just made up the name, sorry. This boss is small but jumps ultra high. Also it takes up many hits to bring down so don't you dare stand in its path to kick/punch. You could a) wait until it reaches the high point in its jump, then as it falls down, attack it, or b) my favorite way. Use the talisman to freeze it then kick it from behind. Rinse and repeat. Your reward? Being congratulated by your assistant and access to the next town =). Don't forget to copy the password. +-------------------------+ |Town 2 - The baby Kyonshi| +-------------------------+ I H S S O O T I O \\* I B |_||_||_||_||_||_||_||_||_||_||_||_| The \\* contains Conshi, the baby Kyonshi. You can have him in your party by letting your assistant be snatched by the light in the graveyard, defeating her (she's very strong, but not too hard), then afterwards you will get the bell. You may want to try it to know how it feels to be a Kyonshi; otherwise do the same modus operandi in Town 1 (learn the Turn Kick as soon as you can), and at the end is another cave... KNIFE KYONSHI (Japanese name: Raunshi) Difficulty: /*** --As soon as you enter, stop for the boss is throwing knives at you. After enough knives, he pauses, that's your signal to get near and whack him. If he hits you with his knife, it does a lot of damage so be careful. You may either use the Tonten to knock him out fast when he is about to use the knives, or the Talisman to freeze him then hit him from behind so he has to turn around to hit you with knives. And if you're wondering whether you can beat this boss using Conshi, well, YOU CAN. It's quite difficult however, and I'm not 100% sure if there are rewards somewhere, but it's quite fun watching him knock out that big guy =). +----------------------------------+ |Town 3 - Woo! A bit of difficulty!| +----------------------------------+ S I I S S T H O S O O B |_||_||_||_||_||_||_||_||_||_||_||_| Save the temple, get some scrolls, learn at the training hall, and then get the three orbs. Simple as that. LIGHTNING KYONSHI Difficulty: */*** --From here, the manual doesn't say the Japanese names, so you have to deal with the translation above. Sorry. --No, he doesn't throw balls of lightning at you. It's that the back- ground flashes every now and then. The boss itself is like any other tall Kyonshi, except he's much faster than the others, much stronger, and much more HP. +-------------------------------------+ |Town 4 - Hey! Be careful! WATCH OUT!!| +-------------------------------------+ H S S O S I O I T O I B |_||_||_||_||_||_||_||_||_||_||_||_| Don't bother learning the Wind Kick; it's pretty much useless. The other move, Mirage Thrust, is a must-have. You can easily kill tall bosses without even getting stratched. I know it's cheap, but an army of kyonshies vs. one phantom fighter isn't fair, too. TACKLING KYONSHI Difficulty: * / * --This is the part where I always got stuck when I was a kid. This boss attacks you at a speed in which you could rarely, if ever, react in time. If you don't hit him with anything in your arsenal, he WILL tackle you. Talisman does not work on him. The Tonten and the Sacred Sword does work on preventing him to attack, but does little to no damage. Try to put him into a corner then attack him whenever he goes floats back up so the chances of him tackling you will be minimized. +--------------------------------------------+ |Town 5 - Who shakes in terror? You, or them?| +--------------------------------------------+ I \\* S I S I T I O O O B |_||_||_||_||_||_||_||_||_||_||_||_| As you notice, there is no training hall. That means it is at the other town, so collect scrolls till you reach the maximum of 99. EARTHQUAKE KYONSHI Difficulty: * */*** --Why the \"earthquake\"? Because when it lands on the ground, it actually shakes it so much that Kenchi falls over. Also, when you're hit by the claws, you are sent to the other side of the room. The good thing? You could use the Mirage Thrust to post a talisman on him, or just plain punch his knees till his life meter runs out. +-------------------------------------------------------------+ |Town 6 - THIS, has gotta be, the weirdest flying kick ever...| +-------------------------------------------------------------+ H T O O O B |_||_||_||_||_||_| The moment you enter, enroll at the training hall and avail of the powerful Jump Kick. It's a shame you would learn it this late in the game. You may choose to save the temple, or if you feel really almighty, just leave it there and snatch the orbs one by one until you can enter the boss lair... AXE KYONSHI Difficulty: * */*** --He does not throw axes at you; it's his melee weapon. He just holds it out in front of him, swinging it once his near enough. Don't punch him; use your feet both for kicking and dancing around him. +-----------------------------------------------------------+ |Town 7 - What?! I thought I could use magic in this game...| +-----------------------------------------------------------+ O O S S S T H S S S O B |_||_||_||_||_||_||_||_||_||_||_||_| No items. Save the last temple, collect scrolls this one last time, then learn 4 Thrust at the last training hall. After three orbs, fight the last boss... FIREBALL KYONSHI Difficulty: */*** --Obviously, he throws fireballs. Very easy to dodge; just jump high then drop behind him. If you learned every skill, you could do it the easy way (crouch then punch) or the exciting way (stand in front of him and flaunt your amazing abilities). Just don't get burned by fireballs, 'cause it HURTS. +------------------------------+ |Town 8 - The stupid final boss| +------------------------------+ O O O FB |_||_||\\_|---|\\_\\_| The --- represents a bridge, and FB is the FINAL BOSS. Are you ready? There are no temples in here, so once you enter a building, you have to make it till the end. There's no turning back. I just hope you really mastered those moves... OBO Difficulty: */*** --First time I said she was easy, I forgot you have to fight an army of assorted Kyonshies to get to her :). --You're fighting, for the second time (the first was a ghost, remember?), a non-kyonshi. Besides teleportation and magic fireballs, she does nothing but stand there, waiting for you to hit her. Unless you are cursed with kyonshi-slow reflexes, she is very disappointing. Sure wish she was a lot more powerful. You may watch the credits roll, but you won't get a new game. Now it's all over. Kyonshies will never bug you again =). --- ## ---> CREDITS My father, for buying me a cartidge a looong time ago (back in 1989); FCI, for making the game; Nintendo, for a really good console that'll last for ages; CJayC, for hosting the guide on his site, GameFAQs; SatelliteGeibor@aol.com, for the Kyonshi names; and you, for playing the game. --- Contributor Recognition: http://www.gamefaqs.com/features/recognition/37592.html --- +------------------------------------------------------------------+","title":"Phantom Fighter"},{"location":"nes/#phantom-fighter-faq","text":"|>>>>>>>>>>>>>>>>>>Copyright 2003 Aaron Madrinan<<<<<<<<<<<<<<<<<<<| +------------------------------------------------------------------+ --- Phantom Fighter FAQ by The Deacon e-mail:mmacinni@cs.oberlin.edu --- Before we start, even though they are probably out of business, the game \"Phantom Fighter\" for the NES is trademark copyright etc. FCI and whoever else made it. Feel free to toss this FAQ to the winds, just don't be a lamer and put your name on it; if you have anything to add, e-mail me and I'll insert your info (as well as credit you with the addendum). ## Contents I.Why is this in FAQ format? I. Why is this in FAQ format? I know the term \"FAQ\" gets bandied about a lot, but since I'm something of a wordsmith I feel that we should nail down exactly what a \"FAQ\" is. You are reading a \"FAQ\", a compilation of \"Frequently Asked Questions\". Christ, if you have net access you should know by now what a FAQ is. Anyhow, it's also a FAQ because walkthroughs are really lame. The way I see it, any kind of aid on a game signals one's inability to rise to the challenge, however in the greater scope things I figure we can call getting level passwords the \"little white cheat\", reading a FAQ a medium or venial cheat, and working the keyboard/mouse with one hand while holding the walkthrough in the other and looking at the screen only to perform the walkthrough's dictated actions is a \"mortal\" cheat. And besides, walkthroughs are really only the kind of thing I could see a reasonable person using in a game with some kind of plot to it. I know a fellow who isn't very good at video games at all, and with as little patience as he has, when he gets a game he gets the walkthrough and goes through the game just to see the plot. However, Phantom Fighter (how's bout we shorten that to \"PF\") has _no_ plot. Really. You'll see. II. What's the plot? I just told you. There isn't one. (see above). However, if you want a basic rundown of the pseudo-plot, well then here you are: You are Kenchi, martial artist and aspiring Phantom Fighter. With your trusty sidekick you travel from town to town seeking fortune and fame as a wandering hero, driving out evil hopping vampires and rescuing innocent townspeople using only your wits, a few sacred items, and your superior kung-fu. That's about the size of it. III. What's the point of the game? Well, as with all games, it's to have fun...:). Besides that, there doesn't really seem to be much of a point...if you were to say that the \"point\" of the game is to be defined as what is required to beat a game, then I suppose there is a point. That definition makes the point of this game thus: You go from town to town (there are 8 in all), driving out vampires with your fists (and feet) of fury. Along the way, you pick up sacred items and ancient scrolls (more on them later) and improve your kung-fu until the final confrontation with the evil witch Obe. (\"obay\") IV. Anything I should know before I start the game? Since I didn't have the manual I'll take the time to explain the basic game mechanics for you. A: Kick. (or, if you prefer, \"Boot to the Head\") B: Punch. Start: Pause the game. Select: Does nothing as far as I know. V. What's the basic pattern of the game? Basically, you'll come to a town with your sidekick in tow. The mayor (I suppose) comes out and kowtows to you, weeping with gladness; for they called the thunder and now they got it. When you enter town, you'll be in a far-off side-scroll view with Kenchi and his sidekick walking around near buildings. When in front of a door, push A or B to see a brief message telling you what's going on inside. (I'll explain them later.) To enter a house, stop in front of the door and press Up. When you enter the house, your energy will be displayed on the left side of the screen, and any enemies in the room will have theirs displayed on the right. As you will soon see, a purple bar means you have full health, and as you take hits the little boxes turn pink until you get wiped out (and your sidekick speaks ill of the dead). In this screen, pressing up makes you jump, down makes you crouch. You cannot attack while crouching. (A BIG PISSER IMHO) Obviously, when you see a vampire come hopping up at you, kick and punch it until it falls over and bursts into flame, then continue to the right. Events vary, but after getting to the end of the house you will be presented with either scrolls or a sacred item, sometimes information (other times just blather). Some houses contain lots of enemies, make your way to the end, defeat a moderately tough enemy, and collect a jewel that instantly refills your health. You'll see it sit one of the empty stands in the upper right corner of the screen. Collect three of these things and you will have the magical power necessary to break a sealed portal. When you have three gems, you can enter the building that is always all the way at the far right end of town. Inside is a town boss, the source of the vampire scourge (for this area, anyhow). Teach him a lesson (\"Don't think. Feeeeeeelll!\") and you'll enjoy being toadied to by your sidekick. Then the happy mayor will give you a password and you're on your way to the next town. VI. What the hell ARE these things I'm fighting? Those, gamer, are hopping vampires -- China's version of the ubiquitous super- natural parasitic entity that has frightened many a superstitious person for hundreds of years. I believe the game's term for them -- \"Kyonshi\" -- is Mandarin Chinese for \"hopping vampire\". However, it could be a bastardized Chinese put into a game for mainly Japanese players. I think in Cantonese the term is \"Jiangshi\". I'll refer to them throughout this document as \"Kyonshi\", simply because that's what the game calls them. What makes these guys different from their Romanian \"cousins\"? Well, their origins are slightly different. Kyonshi are \"undead\", as in those who have died, but somehow have had their souls trapped in their body instead of escaping to the Nine Heavens (or Hells) after death. This could be for any number of reasons; bad chi flow through the house, suicide, lots of angst, etc. Anyhow, because they have died, their bodies aren't alive, they're just \"undead\". This means that they're not as limber as they once were, hence the hopping; it's the only way they can get around. What motivates them? Not much. Unlike the Western vampire, a Kyonshi has little or no conscious thought. Its only thought is to attack living things and eat their bodies. Since they don't consume their victims whole their victims become vampires as well. Kyonshi don't speak, don't respond intelligently to problems. This is a noticable advantage that the phantom fighter has over his unnatural opponents. A nimble phantom fighter should dance about to keep his assailant's long fingernails (so sharp they're almost claws) at bay. For those interested in Kyonshi I'll provide a bibliography at the end of this FAQ -- they're pretty cool. It also helps to know something about them for later (see below). VII. Do the towns follow a basic pattern? Yes. For the most part they are a bunch of houses, a temple, a \"boss shack\", and a martial arts studio. You can always recognize the temple and the studio, they're the same in every town. The temple is a large building with pillars in the front and lots of pillars and Buddha statues inside. The studio is a long, one-level wooden building. There will be three places where you can get gems as well, and the rest of the little shacks round out the town with places to kill Kyonshi and get ancient scrolls. This seems as good a time as any to explain what the little messages you get when you press A or B in front of a house mean. \"Kyonshies are here\": Ahem. \"Kyonshies are not here\":Ahem. Ahem. Okay, besides the obvious, these mean there is just a basic bunch of Kyonshies in the house (or the house is empty). Fight through all of these and you'll get some ancient scrolls. Empty houses seem to have no point; you walk all the way to the end and some cowering peasant thanks you or says something worthy of the term \"non sequitir.\" \"This is a temple.\": What the Christ. It's a temple. \"There's danger in the air\": This means that inside is either a sacred item or one of the three gems you need to bust them ghosts. \"What, nothing's happening?\":or something like that. Basically one of those houses with goodies or gems in them that you've cleared out already. \"Open the sealenter\": Obviously an error in the programming, this should read something like \"Open the seal to enter\". This is the \"boss shack\" that you need the gems to get inside. \"Enter with courage\":When you have the gems, you can open the seal to the \"boss shack\" and instruct the Kyonshi on the finer points of kung fu. (\"I hope you were paying attention.\") VIII. From the looks of things, I'm not a very good phantom fighter. I just got my butt kicked. Any way to improve on this sad situation? Fortunately, yes. At the beginning it may seem like you'll have a lame kick and punch for the entire game, but fear not. You can improve to such powerful kung-fu that you'll easily steamroll enemies. So, how do you go about improving your skills? Easy. First, you have to do a little struggling; you need to earn ancient scrolls. Just pick a small house and go in. When you get some scrolls go to the studio. When you enter, a fat guard accosts you, and asks you if you understand that you won't get kung-fu lessons for free (This is basically to spare you from the annoying un-skippable dialog that follows if you don't want to enter after all). When you say that you understand, he asks you a question, to test your knowledge. A. What the hell is this? Copy protection? I have no idea why this in the game. I can only guess that at the time this was made, it made sense, or really was to protect from people pirating the game. I don't have the manual, so I don't know. Don't sweat it too much, the questions are pretty easy, often it's obvious what the correct answer is, and if you fail you can always try again. B. Fuck that. Just give me the questions and answers. Ok, I guess quiz games aren't your thing. Here are some of the questions that the fat guy asks, and their answers: (These are from memory, more to come) \"Name an FCI video game.\" I think all of these are FCI games, pick \"WCW\" just to be sure. \"Why do Kyonshies only come out at night?\" They hate the sun, obviously. \"How do you capture a Kyonshie?\" Although you never see it in the game, you use an Urn. \"What is a Kyonshies least favorite food?\" I don't know where this hell this came from, by process of elimination it's ice cream. When you answer correctly the fat guy lets you pass and you can enter the Master's chambers, where they gather for the feast. And though they stab it with their steely knives, they just can't kill the beast. Sorry. If you got that reference you'll know the special sort of anguish I feel now at selling my copy of that album. Anyhow, the master will train you in kung fu in exchange for ancient scrolls. Why, I don't know, but my motto is do what works. So, cough up scrolls and train in the martial arts. The different moves you can select are listed with their scroll cost to the left of them. X. What are the different kung-fu moves, how much do they cost, and what do they do? Good question, grasshopper. There are three basic move categories: \"moves\" (\"mv\"):your movement \"thrust\" :punching moves \"kick\" :ahem, kicking moves. Here's a breakdown by category: MOVES: Wolf Move (\"wolf mv.\") Cost: 2 scrolls What it does: Makes you move faster. Useful. Tiger Move (\"tiger mv.\") Cost: 6 scrolls What it does: see above. Mirage Move (\"mirage mv.\") Cost: 6 scrolls What it does: see above. Dragon Move (\"dragon mv.\") Cost: 18 scrolls What it does: see above. Mirage Walk (\"mirage wk.\") Cost: x scrolls What it does: Allows you to walk while crouching. Mirage Thrust (\"mirage th.\") Cost: 50 scrolls What it does: Allows you to punch while crouching. A terrific move. High Jump Cost: x scrolls What it does: you jump higher. Absolutely necessary for airborne vampires. Windmill Jump (\"wind jump\") Cost: x scrolls What it does: you jump still higher, and you flip in the air as you do. Allows you full range with airborne enemies and looks really cool. THRUSTS: 2 Thrust Cost: 2 scrolls What it does: Gives you two lightning-fast punches instead of one. 3 Thrust Cost: x scrolls What it does: Basically same as above, except now it's a 1-2-3 punch. 4 Thrust Cost: 90 scrolls What it does: Now, you punch both high and low, twice. The most powerful move in the game, believe it or not. KICKS: 2 Kick Cost: 6 scrolls What it does: the 1-2 kick. One for both ears. :) Side kick Cost: 18 scrolls What it does: you lean to the side and use your hips to put a little torque action into your kick. Good damage, another move you'll use right up to the end. Windmill Kick (\"wind kick\") Cost: 50 scrolls What it does: When you jump straight up in the air and kick, you'll spin around in a flurry of feet. Really only useful for airborne enemies, and doesn't always come off clean. Jump Kick Cost: 80 scrolls What it does: Run at your enemy and kick and you'll leap at them with both feet forward. (\"Mind your Manners!!!\") Great as an opening move, as well as for enemies who like to jump a lot. XI. What are some good fighting tactics? Stick and move. If you stand in one spot you'll get creamed, as more often than not enemies will soak one or two kicks and swipe you with them claws. This is most important at the beginning of the game, when you can't do anything but punch and kick, and move around a little bit. The main method of attack Kyonshies use is to take little hops at you, then when they get close enough to a big Kyonshie leap into your face. What you want to try to do is to place yourself at a point where the Kyonshie will be in midleap when it comes into kick range. Then, when it leaps, kick it, and it'll fall over. It won't die, but knocking it down gives you breathing room. You'll take a lot of hits from Kyonshies just steadily leaping towards you. That's their other fighting tactic...they just keep coming, soaking up the kicks and punches until they can cut you. If you can kick them enough times, however, they'll fall over. Different kyonshies take more or less kicks/punches in a row to be knocked down, size is one factor. Later, when you learn more kung fu, it will be easier to defeat enemies, although the enemies will get progressively harder as you go from town to town. One good move that will help you out a lot is the \"Wind Jump\". This allows you to leap high into the air (doing a cool kung-fu flip) above the reach of the dreaded hoppping demons. When enemies come at you, kick them once or twice, and if they don't fall down hop over them just before they get to you. A little practice helps, but soon you will be able to drop down right on the other side of them and kick them in the back of the head while they are turning around. Kyonshies are very stupid. It helps a lot to remember this, as the next trick shows. See, Kyonshies (as you will know if you read the above section on them) are in the throes of rigor mortis, their bodies in an arrested state of decay. So they can only see straight forward, right? So if you crouch down, they will turn back and forth, their tiny minds trying to figure out how you pulled your disappearing act. This will not be very useful in the beginning, as you can't walk while crouching. DON'T try this just before a Kyonshie gets to you -- their turning back and forth motion will still be able to hit you, as you are not completely under their arm level (unless you are fighting a really tall kyonshie, see below). This move becomes more useful when you learn the Mirage Walk, as you'll be able to shuffle about under their vision, just don't attempt to get too close to them as they can often still hit you. Later, when you learn Mirage Thrust, this becomes a winning tactic, allowing you to stand under tall enemies and bosses and punch them in their unfeeling vampire gonads mercilessly. (WARNING: This is kind of a cheapo move, as you really can kill tall kyonshi bosses with this tactic. It certainly doesn't _look_ very exciting.) XII. Alright. I'm in town. What do I do first? Wander through town until you find the temple, then try to fight your way through it, shouldn't be to difficult. In the first town, it's one of the first buildings, later on it will be farther into town, or sometimes nonexistent. Anyhow, it's important to start here, so that you can fill up your health, you'll need to often and there's only one other way to do it in the game, and that involves more fighting. Once you've done that, go to the kung-fu studio and see how many scrolls you'll need to learn the various kung-fu moves. The dialog trees can be annoying, but get used to them because they are all through the game. :P Anyhow, leave the studio with the moves you want to learn in mind, and find a house with Kyonshies in it. Then enter and work your way to get the scrolls. Surprisingly (or perhaps not, depending on how well versed you are in \"game logic\"), you can enter the same house again and again, continuously obtaining scrolls. You can get up to 99, but don't bother trying that here unless you are sick in bed with chicken pox or something -- you'll have better oppportunities later on when you recieve lots more scrolls for rescuing townspeople. Learn all the moves you can, when the master tells you you've tapped him out, you can move on. A. Just to make it easier on myself, what moves should I learn first? Since your kung-fu is puny, learn attack moves first. Specifically, get 2 Kick and 2 Thrust. Throughout the game, when you have an opportunity to learn an attack move, learn it, and learn the other moves later. While the \"Move\" moves may seem pointless, they are actually very useful, since being able to move quickly will allow you to dance in an out of a kyonshie's range without getting creamed every time you try to hit them. It's also highly important to learn the jump moves, since they allow you to attack airborne enemies, of which there are 3 in the game, all of them important. While I suppose you could beat the game without all of the moves, I wouldn't recommend it, simply because it wouldn't be as fun, as learning new moves is one of the high points of the game. After you've learned all the moves you care to, head for the houses with the gems in them, collect the gems, and head for the boss shack. Don't forget to write down your password! XIII. Hey, these Kyonshies aren't all alike! Quite right. There are several different types of Kyonshies. Basically, there are four basic types of Kyonshi, differentiated by body type: the little one, the medium one, the tall one, and the big fat one. Within these different morphical categories there's lots of room for variation in the different colors they wear. You'll no doubt become quite familiar with the types as you play the game, and certain ones will become infamous. (\"ARgh! Not more green ones!\") XIV. Okay, so there's different types. What are they like? Without further ado: little: This little bastard is fast and hard to hit. He'll be a toughie when you're duking it out in the first town and don't have awesome kung-fu. His basic weakness is that he can't jump that far, and he's really short, making him easy to hop over. Fortunately he's still just tall enough to kick in the face. medium: Your basic vanilla Kyonshi. Some types jump a lot, some are fast, etc. tall: The tall guy moves ponderously slow, and thanks to his height is hard to jump over. He also jumps very high while attacking as well. Naturally enough this is a big mistake on his part; he jumps so high that you can crouch and let him go over you, then \"boot to the head\" while he's turning around. Later on when you learn the Mirage Thrust the tall guy will be at your mercy as your firsts pound mercilessly into his undead loins, and his turning back and forth schtick won't be able to nail you. fat: This guy is tougher than he looks. He's very strong, such that even though he moves slowly, he packs a real wallop. Also, he doesn't jump that high, but is still tall enough that jumping over him is a bit of a sticky wicket. Your big advantage over him is speed. While he can soak up hits and really wummox you when he gets a hit in, if you play it right you can kick once or twice then get the hell out of the way when the Kyonshi express comes through. XV. I won some kind of item, what does it do? Aha, one of the coveted mystical items that you can use in the game. Don't confuse these with scrolls -- ancient scrolls sound cool but don't do jack except buy you kung-fu. However, the items are quite nice. First, here's how to use an item once you've got it. Whenever you obtain an item, you will see it appear at the top of the screen, in a long box to the left of the stands that hold the three gems you'll need to get out of town. However, make no mistake, you are NOT carrying that item. Your faithful sidekick (who has no name that I could tell) is carrying them. Thus, whenever you enter a building, and you want to use an item, turn back as if to leave. Your boy will pop his head in and ask if you want to split, use an item, or ignore him and get back to butt-kicking. Well, ask for an item, and he'll give you a list: strangely enough, he lists all the items you can get, even if you don't have them yet. Pick one, and it will be outlined by a red square at the top of the screen. This means you are now carrying the item, and will use it instead of punching when you push A. Now, here's all the items, and what they do. Talisman: Freezes enemies in their tracks sometimes. Tonten: Knocks baddies onto their cans with a flash of light. Sword: Knocks guys down, you can also slice 'em with it. Bell: Controls the demon-boy. A note of warning: your items will also take hits for you if you get hit. However, this will also break the item, making it useless. Hey, it's an antique. Keep this in mind when fighting with an item. It's a good idea to grab them, they're nice for keeping enemies at bay, and the Tonten is pretty damn invaluable for beating the game. XVI. I bought one of those Game Genies. Any codes for this game? CODE KEY IN . . . EFFECT . . . 1 VTVKEGSA + KAVKOGNA Start with Sword 2 VTVKEGSA + SAVKOGNA Start with Bell 3 VTVKEGSA + UAVKOGNA Start with Tonten 4 VTVKEGSA + XAVKOGNA Start with Talisman 5 LASKNGAA + VAVKOGNA Start with 3 Scrolls 6 TASKNGAA + VAVKOGNA Start with 6 Scrolls 7 SXSZLUSE Infinite energy 8 OVSZPLSV + PESZZLAA Take less damage when attacked (author's note: This is taken verbatim from Galoob's master list of all Game Genie codes for NES games that they ever published. There may, of course, be some home- grown codes floating about. So all due credit goes to Galoob for the above blurb.) XVII. This isn't the most complete FAQ I've ever read, you know... Sorry. This is mostly from memory. If I get e-mails requesting updates I'll saunter back into the game with my rightfully earned cheat codes and research it fully for those who desire it. And besides, if you think this FAQ is incomplete, check out some of the other (admittedly) sparse fare at TSR's NES FAQ page. For right now I'll call an end to this FAQ, just because I figure this is enough to be written about any NES game that wasn't made by Square. :) I'll send my regards first and foremost to TSR, who's NES page is truly one of the best place for NES info on the 'net, not to mention skilled use of graphics and delightful content like the NES oddity page. Also, here's to the place that posted the ROM I used for this FAQ, Big Daddy's International House of ROMs, THE place to get ROMs on the 'net. In my experience, if he doesn't have it here, it doesn't exist yet (at least for NES, anyhow). I'll also send out greets to John Turk, who's NES Underground Library is probably the most ambitious of all the NES pages I've seen. And finally I'll send shots out to all the other NES pages, all those who've stayed cool despite NESticle's appearance, and to all the fellows on IRC channels like #emuroms, #emulator, and #emu who are out there circulating coolness in less than 200k. :) Also a shot out to my friends on #1980Warez, who have nothing to do with NES ROMs, but are cool anyway. Jibes go to all those who feel the need to charge excessive amounts of money for their emulators. Say what you will about effort and reward, guys, but in the end you're creating something used to do something that's, shall we say, a trifle off-color in the eyes of the law. Jibes also go to those who run IRC channels so huge that no chatting goes on in them, and channels that have nothing but DCC bots in them. Thanks for helping turn IRC into a wasteland, guys. So much have I written for gamers, now give me a drink! The Deacon Restore Page","title":"Phantom Fighter FAQ"},{"location":"nes/#training-hall-quiz-questions-and-answers","text":"+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^+ < ////// // // /// //// // ////// ///// ////////// > < // // // // // // // // // // // // // // // > < ///// ////// /////// // //// // // // // // // > < // // // // // // /// // // // // // // > < // // // // // // // // ///// // // // > < > < ////// // ///// // // ////// ////// ////// > < // // // // // // // // // > < //// // // /// ////// // //// //// > < // // // // // // // // // // > < // // ///// // // // ////// // // > +------------------------------------------------------------------+ * * * * * * Training Hall Quiz Questions and Answers * * * * * * A guide by PinKirby (Pinku_Kirby@yahoo.com) +-----------------+ ----Version 1.0---- +-----------------+ June 9th, 2004 -Began this guide -------------------- Contents: 1. Introduction 2. The Training Hall 3. The Questions and Answers 4. Possible Messages 5. Thanks 6. Legal Info --------------------- * * * * * * 1. Introduction * * * * * * Phantom Fighter is a game that is not very well-known, but is still, in my opinion, a great game. I've decided to make a guide on the questions the guard of the Training Hall asks you. Most questions have one correct answer, but some questions are asked in which ALL of the multiple choices are correct answers. Here's how to use this guide, using a sample question NOT asked in the game: ***QUESTION: What game does this guide cover? 1. Super Mario Brothers (X) 2. Phantom Fighter (O) 3. This game has no name(X) (X) - Means an incorrect answer, in which the guard throws you out. (O) - Means a correct answer, in which the guard lets you pass. --------------------------------------------------------------- * * * * * * 2. The Training Hall * * * * * * After defeating the monsters, the sharp-nailed Chinese phantoms which are known as \"Kyonshies\", in houses and buildings, you can get one or more scrolls. You use these scrolls to learn skills in the Training Hall, but you can't see the master for free, as his guard explains to you. To prove you are worthy of the master's time and attention, you must correctly answer a question with one of three multiple choices. While many of the guard's questions pertain to your foes the Kyonshies, some of the questions involve trivia. If you get a question wrong, DON'T FRET! You can re-enter and try again with another question. The guard tells you, Kenchi: \"You can't see the master for free. Do you understand?\" If you answer no, he'll laugh at your cowardice and kick you out. Answer yes, and it's quiz time! --------------------------------------------------------------- * * * * * * 3. The Questions and Answers * * * * * * QUESTION 1: How many stars are there in the American flag? 1. 50 (O) 2. 28 (X) 3. 5 (X) --------------------- QUESTION 2: What country do samurais come from? 1. India (X) 2. Japan (O) 3. Spain (X) --------------------- QUESTION 3: What is the best method to make sure Kyonshies never revive again? 1. Big Stakes (X) 2. Seal in Rock (X) 3. Fry in Oil (O) --------------------- QUESTION 4: What is the teaching taught by Confucius called? 1. Psychiatry (X) 2. Confucianism (O) 3. Physiognomy (X) --------------------- QUESTION 5: Who built the Great Wall of China? 1. The Emperor (O) 2. Plasterer (O) 3. Farmer (O) --------------------- QUESTION 6: What is Kyonshies' most powerful weapon? 1. Sharp Claws (O) 2. Sharp Eyes (X) 3. Swift Moves (X) --------------------- QUESTION 7: What is the horrible skill called when your head is banged against your opponent's head? 1. Head Strike (O) 2. Head batting (O) 3. Head to Head (O) --------------------- QUESTION 8: What's the name of George Bush's dog? (This is George Bush Sr.!) 1. Millie (O) 2. Martha (X) 3. Pooch (X) --------------------- QUESTION 9: Who created Ultima? 1. Lord British (O) 2. Prince Chuck (X) 3. Mr. Exodus (X) --------------------- QUESTION 10: What is the best thing to use to capture Kyonshies? 1. Coffin (X) 2. Urn (O) 3. Jewelry Box (X) --------------------- QUESTION 11: What is the famous Chinese Newspaper? 1. School Paper (X) 2. Wall Poster (O) 3. Daily China (X) --------------------- QUESTION 11: What is the food Kyonshies hate? 1. Liver (X) 2. Ice Cream (O) 3. Frog Eyes (X) --------------------- QUESTION 12: Something is used to beat Kyonshies. What is it? 1. Bird's Blood (O) 2. Lizard Scale (X) 3. Vulture Nail (X) --------------------- QUESTION 13: Why do Kyonshies only come out at night? 1. Play late (X) 2. Shy (X) 3. Hate the Sun (O) --------------------- QUESTION 14: Which emperor is often called the Last Emperor? 1. Sagi (X) 2. Higi (X) 3. Fugi (O) --------------------- QUESTION 15: What kind of place do Kyoshies usually live in? 1. Wet place (O) 2. New Jersey (O) 3. Beverly Hill (O) --------------------- QUESTION 16: What is the skill called when you hold your opponent's arms from the back and throw him backward? 1. Dragon Suplex (O) 2. Front Suplex (X) 3. Side Suplex (X) --------------------- QUESTION 17: Name an FCI Video Game. 1. Hydlide (O) 2. WCW (O) 3. Ultima (O) --------------------- QUESTION 18: What is a Chinese martial art usually called? 1. Kung Fu (O) 2. Martial Art (X) 3. Phantom Kick (X) --------------------- QUESTION 19: What is another word for one? 1. Uno (O) 2. Ichi (O) 3. Un (O) --------------------- --------------------------------------------------------------- * * * * * * 4. Possible Messages * * * * * * ----- CORRECT MESSAGES: - That's good. The master is waiting for you. Go and see him. If you answer \"New Jersey\" or \"Beverly Hill\" to the question about where Kyonshies usually live in: - That's right. Now as you know, that is the hangout of Kyonshi. Go and enter. (I'm assuming the guard pities you so much that he just accepts it as a correct answer. :P ) ----- INCORRECT MESSAGES: - Why are you trying to pretend you don't know the answer? - You, Kenchi, not know a thing like this? Worth a laugh! Ha ha ha! Now, get out! Get out! - That was a pity. Train yourself and come again. - How little you know I'm stunned. Think it over and come back again. - Kenchi, you are really an idiot! Go home, try to learn more, and come back when you are wiser. If you answer \"5\" as to how many stars are on the US flag: - Dunce! --------------------------------------------------------------- * * * * * * 5. Thanks * * * * * * - To my sister, Mom and Grandpa - To FCI and Pony Canyon for making this game - To YOU for taking the time to read this guide! * * * * * * 6. Legal Stuff * * * * * * This guide is copyrighted (c) by PinKirby. You may copy SOME of this guide, but you MUST give me, PinKirby, credit. If you want to put this guide up on your website, email me and I will see about it. If I find that this guide is on a website NOT listed below, then I will have it removed. SITES THAT MAY HAVE THIS GUIDE UP: www.GameFaqs.com Phatom Fighter is (c) 1989 by FCI and Pony Canyon. The Nintendo Entertainment System and Nintendo of America are registered trademarks of Nintendo. All rights reserved. This Phantom Fighter Training Hall Quiz Guide is (c) by PinKirby Restore Page","title":"Training Hall Quiz Questions and Answers"},{"location":"ollama/","text":"API Endpoints Generate a completion Generate a chat completion Create a Model List Local Models Show Model Information Copy a Model Delete a Model Pull a Model Push a Model Generate Embeddings Conventions Model names Model names follow a model:tag format, where model can have an optional namespace such as example/model . Some examples are orca-mini:3b-q4_1 and llama2:70b . The tag is optional and, if not provided, will default to latest . The tag is used to identify a specific version. Durations All durations are returned in nanoseconds. Streaming responses Certain endpoints stream responses as JSON objects and can optional return non-streamed responses. Generate a completion POST /api/generate Generate a response for a given prompt with a provided model. This is a streaming endpoint, so there will be a series of responses. The final response object will include statistics and additional data from the request. Parameters model : (required) the model name prompt : the prompt to generate a response for images : (optional) a list of base64-encoded images (for multimodal models such as llava ) Advanced parameters (optional): format : the format to return a response in. Currently the only accepted value is json options : additional model parameters listed in the documentation for the Modelfile such as temperature system : system message to (overrides what is defined in the Modelfile ) template : the prompt template to use (overrides what is defined in the Modelfile ) context : the context parameter returned from a previous request to /generate , this can be used to keep a short conversational memory stream : if false the response will be returned as a single response object, rather than a stream of objects raw : if true no formatting will be applied to the prompt. You may choose to use the raw parameter if you are specifying a full templated prompt in your request to the API keep_alive : controls how long the model will stay loaded into memory following the request (default: 5m ) JSON mode Enable JSON mode by setting the format parameter to json . This will structure the response as a valid JSON object. See the JSON mode example below. Note: it's important to instruct the model to use JSON in the prompt . Otherwise, the model may generate large amounts whitespace. Examples Generate request (Streaming) Request curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\", \"prompt\": \"Why is the sky blue?\" }' Response A stream of JSON objects is returned: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T08:52:19.385406455-07:00\", \"response\": \"The\", \"done\": false } The final response in the stream also includes additional data about the generation: total_duration : time spent generating the response load_duration : time spent in nanoseconds loading the model prompt_eval_count : number of tokens in the prompt prompt_eval_duration : time spent in nanoseconds evaluating the prompt eval_count : number of tokens the response eval_duration : time in nanoseconds spent generating the response context : an encoding of the conversation used in this response, this can be sent in the next request to keep a conversational memory response : empty if the response was streamed, if not streamed, this will contain the full response To calculate how fast the response is generated in tokens per second (token/s), divide eval_count / eval_duration . { \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"response\": \"\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 10706818083, \"load_duration\": 6338219291, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 130079000, \"eval_count\": 259, \"eval_duration\": 4232710000 } Request (No streaming) Request A response can be received in one reply when streaming is off. curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\", \"prompt\": \"Why is the sky blue?\", \"stream\": false }' Response If stream is set to false , the response will be a single JSON object: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"response\": \"The sky is blue because it is the color of the sky.\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 5043500667, \"load_duration\": 5025959, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 325953000, \"eval_count\": 290, \"eval_duration\": 4709213000 } Request (JSON mode) When format is set to json , the output will always be a well-formed JSON object. It's important to also instruct the model to respond in JSON. Request curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\", \"prompt\": \"What color is the sky at different times of the day? Respond using JSON\", \"format\": \"json\", \"stream\": false }' Response { \"model\": \"llama2\", \"created_at\": \"2023-11-09T21:07:55.186497Z\", \"response\": \"{\\n\\\"morning\\\": {\\n\\\"color\\\": \\\"blue\\\"\\n},\\n\\\"noon\\\": {\\n\\\"color\\\": \\\"blue-gray\\\"\\n},\\n\\\"afternoon\\\": {\\n\\\"color\\\": \\\"warm gray\\\"\\n},\\n\\\"evening\\\": {\\n\\\"color\\\": \\\"orange\\\"\\n}\\n}\\n\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 4648158584, \"load_duration\": 4071084, \"prompt_eval_count\": 36, \"prompt_eval_duration\": 439038000, \"eval_count\": 180, \"eval_duration\": 4196918000 } The value of response will be a string containing JSON similar to: { \"morning\": { \"color\": \"blue\" }, \"noon\": { \"color\": \"blue-gray\" }, \"afternoon\": { \"color\": \"warm gray\" }, \"evening\": { \"color\": \"orange\" } } Request (with images) To submit images to multimodal models such as llava or bakllava , provide a list of base64-encoded images : Request curl http://localhost:11434/api/generate -d '{ \"model\": \"llava\", \"prompt\":\"What is in this picture?\", \"stream\": false, \"images\": [\"iVBORw0KGgoAAAANSUhEUgAAAG0AAABmCAYAAADBPx+VAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAA3VSURBVHgB7Z27r0zdG8fX743i1bi1ikMoFMQloXRpKFFIqI7LH4BEQ+NWIkjQuSWCRIEoULk0gsK1kCBI0IhrQVT7tz/7zZo888yz1r7MnDl7z5xvsjkzs2fP3uu71nNfa7lkAsm7d++Sffv2JbNmzUqcc8m0adOSzZs3Z+/XES4ZckAWJEGWPiCxjsQNLWmQsWjRIpMseaxcuTKpG/7HP27I8P79e7dq1ars/yL4/v27S0ejqwv+cUOGEGGpKHR37tzJCEpHV9tnT58+dXXCJDdECBE2Ojrqjh071hpNECjx4cMHVycM1Uhbv359B2F79+51586daxN/+pyRkRFXKyRDAqxEp4yMlDDzXG1NPnnyJKkThoK0VFd1ELZu3TrzXKxKfW7dMBQ6bcuWLW2v0VlHjx41z717927ba22U9APcw7Nnz1oGEPeL3m3p2mTAYYnFmMOMXybPPXv2bNIPpFZr1NHn4HMw0KRBjg9NuRw95s8PEcz/6DZELQd/09C9QGq5RsmSRybqkwHGjh07OsJSsYYm3ijPpyHzoiacg35MLdDSIS/O1yM778jOTwYUkKNHWUzUWaOsylE00MyI0fcnOwIdjvtNdW/HZwNLGg+sR1kMepSNJXmIwxBZiG8tDTpEZzKg0GItNsosY8USkxDhD0Rinuiko2gfL/RbiD2LZAjU9zKQJj8RDR0vJBR1/Phx9+PHj9Z7REF4nTZkxzX4LCXHrV271qXkBAPGfP/atWvu/PnzHe4C97F48eIsRLZ9+3a3f/9+87dwP1JxaF7/3r17ba+5l4EcaVo0lj3SBq5kGTJSQmLWMjgYNei2GPT1MuMqGTDEFHzeQSP2wi/jGnkmPJ/nhccs44jvDAxpVcxnq0F6eT8h4ni/iIWpR5lPyA6ETkNXoSukvpJAD3AsXLiwpZs49+fPn5ke4j10TqYvegSfn0OnafC+Tv9ooA/JPkgQysqQNBzagXY55nO/oa1F7qvIPWkRL12WRpMWUvpVDYmxAPehxWSe8ZEXL20sadYIozfmNch4QJPAfeJgW3rNsnzphBKNJM2KKODo1rVOMRYik5ETy3ix4qWNI81qAAirizgMIc+yhTytx0JWZuNI03qsrgWlGtwjoS9XwgUhWGyhUaRZZQNNIEwCiXD16tXcAHUs79co0vSD8rrJCIW98pzvxpAWyyo3HYwqS0+H0BjStClcZJT5coMm6D2LOF8TolGJtK9fvyZpyiC5ePFi9nc/oJU4eiEP0jVoAnHa9wyJycITMP78+eMeP37sXrx44d6+fdt6f82aNdkx1pg9e3Zb5W+RSRE+n+VjksQWifvVaTKFhn5O8my63K8Qabdv33b379/PiAP//vuvW7BggZszZ072/+TJk91YgkafPn166zXB1rQHFvouAWHq9z3SEevSUerqCn2/dDCeta2jxYbr69evk4MHDyY7d+7MjhMnTiTPnz9Pfv/+nfQT2ggpO2dMF8cghuoM7Ygj5iWCqRlGFml0QC/ftGmTmzt3rmsaKDsgBSPh0/8yPeLLBihLkOKJc0jp8H8vUzcxIA1k6QJ/c78tWEyj5P3o4u9+jywNPdJi5rAH9x0KHcl4Hg570eQp3+vHXGyrmEeigzQsQsjavXt38ujRo44LQuDDhw+TW7duRS1HGgMxhNXHgflaNTOsHyKvHK5Ijo2jbFjJBQK9YwFd6RVMzfgRBmEfP37suBBm/p49e1qjEP2mwTViNRo0VJWH1deMXcNK08uUjVUu7s/zRaL+oLNxz1bpANco4npUgX4G2eFbpDFyQoQxojBCpEGSytmOH8qrH5Q9vuzD6ofQylkCUmh8DBAr+q8JCyVNtWQIidKQE9wNtLSQnS4jDSsxNHogzFuQBw4cyM61UKVsjfr3ooBkPSqqQHesUPWVtzi9/vQi1T+rJj7WiTz4Pt/l3LxUkr5P2VYZaZ4URpsE+st/dujQoaBBYokbrz/8TJNQYLSonrPS9kUaSkPeZyj1AWSj+d+VBoy1pIWVNed8P0Ll/ee5HdGRhrHhR5GGN0r4LGZBaj8oFDJitBTJzIZgFcmU0Y8ytWMZMzJOaXUSrUs5RxKnrxmbb5YXO9VGUhtpXldhEUogFr3IzIsvlpmdosVcGVGXFWp2oU9kLFL3dEkSz6NHEY1sjSRdIuDFWEhd8KxFqsRi1uM/nz9/zpxnwlESONdg6dKlbsaMGS4EHFHtjFIDHwKOo46l4TxSuxgDzi+rE2jg+BaFruOX4HXa0Nnf1lwAPufZeF8/r6zD97WK2qFnGjBxTw5qNGPxT+5T/r7/7RawFC3j4vTp09koCxkeHjqbHJqArmH5UrFKKksnxrK7FuRIs8STfBZv+luugXZ2pR/pP9Ois4z+TiMzUUkUjD0iEi1fzX8GmXyuxUBRcaUfykV0YZnlJGKQpOiGB76x5GeWkWWJc3mOrK6S7xdND+W5N6XyaRgtWJFe13GkaZnKOsYqGdOVVVbGupsyA/l7emTLHi7vwTdirNEt0qxnzAvBFcnQF16xh/TMpUuXHDowhlA9vQVraQhkudRdzOnK+04ZSP3DUhVSP61YsaLtd/ks7ZgtPcXqPqEafHkdqa84X6aCeL7YWlv6edGFHb+ZFICPlljHhg0bKuk0CSvVznWsotRu433alNdFrqG45ejoaPCaUkWERpLXjzFL2Rpllp7PJU2a/v7Ab8N05/9t27Z16KUqoFGsxnI9EosS2niSYg9SpU6B4JgTrvVW1flt1sT+0ADIJU2maXzcUTraGCRaL1Wp9rUMk16PMom8QhruxzvZIegJjFU7LLCePfS8uaQdPny4jTTL0dbee5mYokQsXTIWNY46kuMbnt8Kmec+LGWtOVIl9cT1rCB0V8WqkjAsRwta93TbwNYoGKsUSChN44lgBNCoHLHzquYKrU6qZ8lolCIN0Rh6cP0Q3U6I6IXILYOQI513hJaSKAorFpuHXJNfVlpRtmYBk1Su1obZr5dnKAO+L10Hrj3WZW+E3qh6IszE37F6EB+68mGpvKm4eb9bFrlzrok7fvr0Kfv727dvWRmdVTJHw0qiiCUSZ6wCK+7XL/AcsgNyL74DQQ730sv78Su7+t/A36MdY0sW5o40ahslXr58aZ5HtZB8GH64m9EmMZ7FpYw4T6QnrZfgenrhFxaSiSGXtPnz57e9TkNZLvTjeqhr734CNtrK41L40sUQckmj1lGKQ0rC37x544r8eNXRpnVE3ZZY7zXo8NomiO0ZUCj2uHz58rbXoZ6gc0uA+F6ZeKS/jhRDUq8MKrTho9fEkihMmhxtBI1DxKFY9XLpVcSkfoi8JGnToZO5sU5aiDQIW716ddt7ZLYtMQlhECdBGXZZMWldY5BHm5xgAroWj4C0hbYkSc/jBmggIrXJWlZM6pSETsEPGqZOndr2uuuR5rF169a2HoHPdurUKZM4CO1WTPqaDaAd+GFGKdIQkxAn9RuEWcTRyN2KSUgiSgF5aWzPTeA/lN5rZubMmR2bE4SIC4nJoltgAV/dVefZm72AtctUCJU2CMJ327hxY9t7EHbkyJFseq+EJSY16RPo3Dkq1kkr7+q0bNmyDuLQcZBEPYmHVdOBiJyIlrRDq41YPWfXOxUysi5fvtyaj+2BpcnsUV/oSoEMOk2CQGlr4ckhBwaetBhjCwH0ZHtJROPJkyc7UjcYLDjmrH7ADTEBXFfOYmB0k9oYBOjJ8b4aOYSe7QkKcYhFlq3QYLQhSidNmtS2RATwy8YOM3EQJsUjKiaWZ+vZToUQgzhkHXudb/PW5YMHD9yZM2faPsMwoc7RciYJXbGuBqJ1UIGKKLv915jsvgtJxCZDubdXr165mzdvtr1Hz5LONA8jrUwKPqsmVesKa49S3Q4WxmRPUEYdTjgiUcfUwLx589ySJUva3oMkP6IYddq6HMS4o55xBJBUeRjzfa4Zdeg56QZ43LhxoyPo7Lf1kNt7oO8wWAbNwaYjIv5lhyS7kRf96dvm5Jah8vfvX3flyhX35cuX6HfzFHOToS1H4BenCaHvO8pr8iDuwoUL7tevX+b5ZdbBair0xkFIlFDlW4ZknEClsp/TzXyAKVOmmHWFVSbDNw1l1+4f90U6IY/q4V27dpnE9bJ+v87QEydjqx/UamVVPRG+mwkNTYN+9tjkwzEx+atCm/X9WvWtDtAb68Wy9LXa1UmvCDDIpPkyOQ5ZwSzJ4jMrvFcr0rSjOUh+GcT4LSg5ugkW1Io0/SCDQBojh0hPlaJdah+tkVYrnTZowP8iq1F1TgMBBauufyB33x1v+NWFYmT5KmppgHC+NkAgbmRkpD3yn9QIseXymoTQFGQmIOKTxiZIWpvAatenVqRVXf2nTrAWMsPnKrMZHz6bJq5jvce6QK8J1cQNgKxlJapMPdZSR64/UivS9NztpkVEdKcrs5alhhWP9NeqlfWopzhZScI6QxseegZRGeg5a8C3Re1Mfl1ScP36ddcUaMuv24iOJtz7sbUjTS4qBvKmstYJoUauiuD3k5qhyr7QdUHMeCgLa1Ear9NquemdXgmum4fvJ6w1lqsuDhNrg1qSpleJK7K3TF0Q2jSd94uSZ60kK1e3qyVpQK6PVWXp2/FC3mp6jBhKKOiY2h3gtUV64TWM6wDETRPLDfSakXmH3w8g9Jlug8ZtTt4kVF0kLUYYmCCtD/DrQ5YhMGbA9L3ucdjh0y8kOHW5gU/VEEmJTcL4Pz/f7mgoAbYkAAAAAElFTkSuQmCC\"] }' Response { \"model\": \"llava\", \"created_at\": \"2023-11-03T15:36:02.583064Z\", \"response\": \"A happy cartoon character, which is cute and cheerful.\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 2938432250, \"load_duration\": 2559292, \"prompt_eval_count\": 1, \"prompt_eval_duration\": 2195557000, \"eval_count\": 44, \"eval_duration\": 736432000 } Request (Raw Mode) In some cases, you may wish to bypass the templating system and provide a full prompt. In this case, you can use the raw parameter to disable templating. Also note that raw mode will not return a context. Request curl http://localhost:11434/api/generate -d '{ \"model\": \"mistral\", \"prompt\": \"[INST] why is the sky blue? [/INST]\", \"raw\": true, \"stream\": false }' Request (Reproducible outputs) For reproducible outputs, set temperature to 0 and seed to a number: Request curl http://localhost:11434/api/generate -d '{ \"model\": \"mistral\", \"prompt\": \"Why is the sky blue?\", \"options\": { \"seed\": 123, \"temperature\": 0 } }' Response { \"model\": \"mistral\", \"created_at\": \"2023-11-03T15:36:02.583064Z\", \"response\": \" The sky appears blue because of a phenomenon called Rayleigh scattering.\", \"done\": true, \"total_duration\": 8493852375, \"load_duration\": 6589624375, \"prompt_eval_count\": 14, \"prompt_eval_duration\": 119039000, \"eval_count\": 110, \"eval_duration\": 1779061000 } Generate request (With options) If you want to set custom options for the model at runtime rather than in the Modelfile, you can do so with the options parameter. This example sets every available option, but you can set any of them individually and omit the ones you do not want to override. Request curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\", \"prompt\": \"Why is the sky blue?\", \"stream\": false, \"options\": { \"num_keep\": 5, \"seed\": 42, \"num_predict\": 100, \"top_k\": 20, \"top_p\": 0.9, \"tfs_z\": 0.5, \"typical_p\": 0.7, \"repeat_last_n\": 33, \"temperature\": 0.8, \"repeat_penalty\": 1.2, \"presence_penalty\": 1.5, \"frequency_penalty\": 1.0, \"mirostat\": 1, \"mirostat_tau\": 0.8, \"mirostat_eta\": 0.6, \"penalize_newline\": true, \"stop\": [\"\\n\", \"user:\"], \"numa\": false, \"num_ctx\": 1024, \"num_batch\": 2, \"num_gqa\": 1, \"num_gpu\": 1, \"main_gpu\": 0, \"low_vram\": false, \"f16_kv\": true, \"vocab_only\": false, \"use_mmap\": true, \"use_mlock\": false, \"rope_frequency_base\": 1.1, \"rope_frequency_scale\": 0.8, \"num_thread\": 8 } }' Response { \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"response\": \"The sky is blue because it is the color of the sky.\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 4935886791, \"load_duration\": 534986708, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 107345000, \"eval_count\": 237, \"eval_duration\": 4289432000 } Load a model If an empty prompt is provided, the model will be loaded into memory. Request curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\" }' Response A single JSON object is returned: { \"model\": \"llama2\", \"created_at\": \"2023-12-18T19:52:07.071755Z\", \"response\": \"\", \"done\": true } Generate a chat completion POST /api/chat Generate the next message in a chat with a provided model. This is a streaming endpoint, so there will be a series of responses. Streaming can be disabled using \"stream\": false . The final response object will include statistics and additional data from the request. Parameters model : (required) the model name messages : the messages of the chat, this can be used to keep a chat memory The message object has the following fields: role : the role of the message, either system , user or assistant content : the content of the message images (optional): a list of images to include in the message (for multimodal models such as llava ) Advanced parameters (optional): format : the format to return a response in. Currently the only accepted value is json options : additional model parameters listed in the documentation for the Modelfile such as temperature template : the prompt template to use (overrides what is defined in the Modelfile ) stream : if false the response will be returned as a single response object, rather than a stream of objects keep_alive : controls how long the model will stay loaded into memory following the request (default: 5m ) Examples Chat Request (Streaming) Request Send a chat message with a streaming response. curl http://localhost:11434/api/chat -d '{ \"model\": \"llama2\", \"messages\": [ { \"role\": \"user\", \"content\": \"why is the sky blue?\" } ] }' Response A stream of JSON objects is returned: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T08:52:19.385406455-07:00\", \"message\": { \"role\": \"assistant\", \"content\": \"The\", \"images\": null }, \"done\": false } Final response: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"done\": true, \"total_duration\": 4883583458, \"load_duration\": 1334875, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 342546000, \"eval_count\": 282, \"eval_duration\": 4535599000 } Chat request (No streaming) Request curl http://localhost:11434/api/chat -d '{ \"model\": \"llama2\", \"messages\": [ { \"role\": \"user\", \"content\": \"why is the sky blue?\" } ], \"stream\": false }' Response { \"model\": \"registry.ollama.ai/library/llama2:latest\", \"created_at\": \"2023-12-12T14:13:43.416799Z\", \"message\": { \"role\": \"assistant\", \"content\": \"Hello! How are you today?\" }, \"done\": true, \"total_duration\": 5191566416, \"load_duration\": 2154458, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 383809000, \"eval_count\": 298, \"eval_duration\": 4799921000 } Chat request (With History) Send a chat message with a conversation history. You can use this same approach to start the conversation using multi-shot or chain-of-thought prompting. Request curl http://localhost:11434/api/chat -d '{ \"model\": \"llama2\", \"messages\": [ { \"role\": \"user\", \"content\": \"why is the sky blue?\" }, { \"role\": \"assistant\", \"content\": \"due to rayleigh scattering.\" }, { \"role\": \"user\", \"content\": \"how is that different than mie scattering?\" } ] }' Response A stream of JSON objects is returned: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T08:52:19.385406455-07:00\", \"message\": { \"role\": \"assistant\", \"content\": \"The\" }, \"done\": false } Final response: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"done\": true, \"total_duration\": 8113331500, \"load_duration\": 6396458, \"prompt_eval_count\": 61, \"prompt_eval_duration\": 398801000, \"eval_count\": 468, \"eval_duration\": 7701267000 } Chat request (with images) Request Send a chat message with a conversation history. curl http://localhost:11434/api/chat -d '{ \"model\": \"llava\", \"messages\": [ { \"role\": \"user\", \"content\": \"what is in this image?\", \"images\": [\"iVBORw0KGgoAAAANSUhEUgAAAG0AAABmCAYAAADBPx+VAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAA3VSURBVHgB7Z27r0zdG8fX743i1bi1ikMoFMQloXRpKFFIqI7LH4BEQ+NWIkjQuSWCRIEoULk0gsK1kCBI0IhrQVT7tz/7zZo888yz1r7MnDl7z5xvsjkzs2fP3uu71nNfa7lkAsm7d++Sffv2JbNmzUqcc8m0adOSzZs3Z+/XES4ZckAWJEGWPiCxjsQNLWmQsWjRIpMseaxcuTKpG/7HP27I8P79e7dq1ars/yL4/v27S0ejqwv+cUOGEGGpKHR37tzJCEpHV9tnT58+dXXCJDdECBE2Ojrqjh071hpNECjx4cMHVycM1Uhbv359B2F79+51586daxN/+pyRkRFXKyRDAqxEp4yMlDDzXG1NPnnyJKkThoK0VFd1ELZu3TrzXKxKfW7dMBQ6bcuWLW2v0VlHjx41z717927ba22U9APcw7Nnz1oGEPeL3m3p2mTAYYnFmMOMXybPPXv2bNIPpFZr1NHn4HMw0KRBjg9NuRw95s8PEcz/6DZELQd/09C9QGq5RsmSRybqkwHGjh07OsJSsYYm3ijPpyHzoiacg35MLdDSIS/O1yM778jOTwYUkKNHWUzUWaOsylE00MyI0fcnOwIdjvtNdW/HZwNLGg+sR1kMepSNJXmIwxBZiG8tDTpEZzKg0GItNsosY8USkxDhD0Rinuiko2gfL/RbiD2LZAjU9zKQJj8RDR0vJBR1/Phx9+PHj9Z7REF4nTZkxzX4LCXHrV271qXkBAPGfP/atWvu/PnzHe4C97F48eIsRLZ9+3a3f/9+87dwP1JxaF7/3r17ba+5l4EcaVo0lj3SBq5kGTJSQmLWMjgYNei2GPT1MuMqGTDEFHzeQSP2wi/jGnkmPJ/nhccs44jvDAxpVcxnq0F6eT8h4ni/iIWpR5lPyA6ETkNXoSukvpJAD3AsXLiwpZs49+fPn5ke4j10TqYvegSfn0OnafC+Tv9ooA/JPkgQysqQNBzagXY55nO/oa1F7qvIPWkRL12WRpMWUvpVDYmxAPehxWSe8ZEXL20sadYIozfmNch4QJPAfeJgW3rNsnzphBKNJM2KKODo1rVOMRYik5ETy3ix4qWNI81qAAirizgMIc+yhTytx0JWZuNI03qsrgWlGtwjoS9XwgUhWGyhUaRZZQNNIEwCiXD16tXcAHUs79co0vSD8rrJCIW98pzvxpAWyyo3HYwqS0+H0BjStClcZJT5coMm6D2LOF8TolGJtK9fvyZpyiC5ePFi9nc/oJU4eiEP0jVoAnHa9wyJycITMP78+eMeP37sXrx44d6+fdt6f82aNdkx1pg9e3Zb5W+RSRE+n+VjksQWifvVaTKFhn5O8my63K8Qabdv33b379/PiAP//vuvW7BggZszZ072/+TJk91YgkafPn166zXB1rQHFvouAWHq9z3SEevSUerqCn2/dDCeta2jxYbr69evk4MHDyY7d+7MjhMnTiTPnz9Pfv/+nfQT2ggpO2dMF8cghuoM7Ygj5iWCqRlGFml0QC/ftGmTmzt3rmsaKDsgBSPh0/8yPeLLBihLkOKJc0jp8H8vUzcxIA1k6QJ/c78tWEyj5P3o4u9+jywNPdJi5rAH9x0KHcl4Hg570eQp3+vHXGyrmEeigzQsQsjavXt38ujRo44LQuDDhw+TW7duRS1HGgMxhNXHgflaNTOsHyKvHK5Ijo2jbFjJBQK9YwFd6RVMzfgRBmEfP37suBBm/p49e1qjEP2mwTViNRo0VJWH1deMXcNK08uUjVUu7s/zRaL+oLNxz1bpANco4npUgX4G2eFbpDFyQoQxojBCpEGSytmOH8qrH5Q9vuzD6ofQylkCUmh8DBAr+q8JCyVNtWQIidKQE9wNtLSQnS4jDSsxNHogzFuQBw4cyM61UKVsjfr3ooBkPSqqQHesUPWVtzi9/vQi1T+rJj7WiTz4Pt/l3LxUkr5P2VYZaZ4URpsE+st/dujQoaBBYokbrz/8TJNQYLSonrPS9kUaSkPeZyj1AWSj+d+VBoy1pIWVNed8P0Ll/ee5HdGRhrHhR5GGN0r4LGZBaj8oFDJitBTJzIZgFcmU0Y8ytWMZMzJOaXUSrUs5RxKnrxmbb5YXO9VGUhtpXldhEUogFr3IzIsvlpmdosVcGVGXFWp2oU9kLFL3dEkSz6NHEY1sjSRdIuDFWEhd8KxFqsRi1uM/nz9/zpxnwlESONdg6dKlbsaMGS4EHFHtjFIDHwKOo46l4TxSuxgDzi+rE2jg+BaFruOX4HXa0Nnf1lwAPufZeF8/r6zD97WK2qFnGjBxTw5qNGPxT+5T/r7/7RawFC3j4vTp09koCxkeHjqbHJqArmH5UrFKKksnxrK7FuRIs8STfBZv+luugXZ2pR/pP9Ois4z+TiMzUUkUjD0iEi1fzX8GmXyuxUBRcaUfykV0YZnlJGKQpOiGB76x5GeWkWWJc3mOrK6S7xdND+W5N6XyaRgtWJFe13GkaZnKOsYqGdOVVVbGupsyA/l7emTLHi7vwTdirNEt0qxnzAvBFcnQF16xh/TMpUuXHDowhlA9vQVraQhkudRdzOnK+04ZSP3DUhVSP61YsaLtd/ks7ZgtPcXqPqEafHkdqa84X6aCeL7YWlv6edGFHb+ZFICPlljHhg0bKuk0CSvVznWsotRu433alNdFrqG45ejoaPCaUkWERpLXjzFL2Rpllp7PJU2a/v7Ab8N05/9t27Z16KUqoFGsxnI9EosS2niSYg9SpU6B4JgTrvVW1flt1sT+0ADIJU2maXzcUTraGCRaL1Wp9rUMk16PMom8QhruxzvZIegJjFU7LLCePfS8uaQdPny4jTTL0dbee5mYokQsXTIWNY46kuMbnt8Kmec+LGWtOVIl9cT1rCB0V8WqkjAsRwta93TbwNYoGKsUSChN44lgBNCoHLHzquYKrU6qZ8lolCIN0Rh6cP0Q3U6I6IXILYOQI513hJaSKAorFpuHXJNfVlpRtmYBk1Su1obZr5dnKAO+L10Hrj3WZW+E3qh6IszE37F6EB+68mGpvKm4eb9bFrlzrok7fvr0Kfv727dvWRmdVTJHw0qiiCUSZ6wCK+7XL/AcsgNyL74DQQ730sv78Su7+t/A36MdY0sW5o40ahslXr58aZ5HtZB8GH64m9EmMZ7FpYw4T6QnrZfgenrhFxaSiSGXtPnz57e9TkNZLvTjeqhr734CNtrK41L40sUQckmj1lGKQ0rC37x544r8eNXRpnVE3ZZY7zXo8NomiO0ZUCj2uHz58rbXoZ6gc0uA+F6ZeKS/jhRDUq8MKrTho9fEkihMmhxtBI1DxKFY9XLpVcSkfoi8JGnToZO5sU5aiDQIW716ddt7ZLYtMQlhECdBGXZZMWldY5BHm5xgAroWj4C0hbYkSc/jBmggIrXJWlZM6pSETsEPGqZOndr2uuuR5rF169a2HoHPdurUKZM4CO1WTPqaDaAd+GFGKdIQkxAn9RuEWcTRyN2KSUgiSgF5aWzPTeA/lN5rZubMmR2bE4SIC4nJoltgAV/dVefZm72AtctUCJU2CMJ327hxY9t7EHbkyJFseq+EJSY16RPo3Dkq1kkr7+q0bNmyDuLQcZBEPYmHVdOBiJyIlrRDq41YPWfXOxUysi5fvtyaj+2BpcnsUV/oSoEMOk2CQGlr4ckhBwaetBhjCwH0ZHtJROPJkyc7UjcYLDjmrH7ADTEBXFfOYmB0k9oYBOjJ8b4aOYSe7QkKcYhFlq3QYLQhSidNmtS2RATwy8YOM3EQJsUjKiaWZ+vZToUQgzhkHXudb/PW5YMHD9yZM2faPsMwoc7RciYJXbGuBqJ1UIGKKLv915jsvgtJxCZDubdXr165mzdvtr1Hz5LONA8jrUwKPqsmVesKa49S3Q4WxmRPUEYdTjgiUcfUwLx589ySJUva3oMkP6IYddq6HMS4o55xBJBUeRjzfa4Zdeg56QZ43LhxoyPo7Lf1kNt7oO8wWAbNwaYjIv5lhyS7kRf96dvm5Jah8vfvX3flyhX35cuX6HfzFHOToS1H4BenCaHvO8pr8iDuwoUL7tevX+b5ZdbBair0xkFIlFDlW4ZknEClsp/TzXyAKVOmmHWFVSbDNw1l1+4f90U6IY/q4V27dpnE9bJ+v87QEydjqx/UamVVPRG+mwkNTYN+9tjkwzEx+atCm/X9WvWtDtAb68Wy9LXa1UmvCDDIpPkyOQ5ZwSzJ4jMrvFcr0rSjOUh+GcT4LSg5ugkW1Io0/SCDQBojh0hPlaJdah+tkVYrnTZowP8iq1F1TgMBBauufyB33x1v+NWFYmT5KmppgHC+NkAgbmRkpD3yn9QIseXymoTQFGQmIOKTxiZIWpvAatenVqRVXf2nTrAWMsPnKrMZHz6bJq5jvce6QK8J1cQNgKxlJapMPdZSR64/UivS9NztpkVEdKcrs5alhhWP9NeqlfWopzhZScI6QxseegZRGeg5a8C3Re1Mfl1ScP36ddcUaMuv24iOJtz7sbUjTS4qBvKmstYJoUauiuD3k5qhyr7QdUHMeCgLa1Ear9NquemdXgmum4fvJ6w1lqsuDhNrg1qSpleJK7K3TF0Q2jSd94uSZ60kK1e3qyVpQK6PVWXp2/FC3mp6jBhKKOiY2h3gtUV64TWM6wDETRPLDfSakXmH3w8g9Jlug8ZtTt4kVF0kLUYYmCCtD/DrQ5YhMGbA9L3ucdjh0y8kOHW5gU/VEEmJTcL4Pz/f7mgoAbYkAAAAAElFTkSuQmCC\"] } ] }' Response { \"model\": \"llava\", \"created_at\": \"2023-12-13T22:42:50.203334Z\", \"message\": { \"role\": \"assistant\", \"content\": \" The image features a cute, little pig with an angry facial expression. It's wearing a heart on its shirt and is waving in the air. This scene appears to be part of a drawing or sketching project.\", \"images\": null }, \"done\": true, \"total_duration\": 1668506709, \"load_duration\": 1986209, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 359682000, \"eval_count\": 83, \"eval_duration\": 1303285000 } Chat request (Reproducible outputs) Request curl http://localhost:11434/api/chat -d '{ \"model\": \"llama2\", \"messages\": [ { \"role\": \"user\", \"content\": \"Hello!\" } ], \"options\": { \"seed\": 101, \"temperature\": 0 } }' Response { \"model\": \"registry.ollama.ai/library/llama2:latest\", \"created_at\": \"2023-12-12T14:13:43.416799Z\", \"message\": { \"role\": \"assistant\", \"content\": \"Hello! How are you today?\" }, \"done\": true, \"total_duration\": 5191566416, \"load_duration\": 2154458, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 383809000, \"eval_count\": 298, \"eval_duration\": 4799921000 } Create a Model POST /api/create Create a model from a Modelfile . It is recommended to set modelfile to the content of the Modelfile rather than just set path . This is a requirement for remote create. Remote model creation must also create any file blobs, fields such as FROM and ADAPTER , explicitly with the server using Create a Blob and the value to the path indicated in the response. Parameters name : name of the model to create modelfile (optional): contents of the Modelfile stream : (optional) if false the response will be returned as a single response object, rather than a stream of objects path (optional): path to the Modelfile Examples Create a new model Create a new model from a Modelfile . Request curl http://localhost:11434/api/create -d '{ \"name\": \"mario\", \"modelfile\": \"FROM llama2\\nSYSTEM You are mario from Super Mario Bros.\" }' Response A stream of JSON objects. Notice that the final JSON object shows a \"status\": \"success\" . {\"status\":\"reading model metadata\"} {\"status\":\"creating system layer\"} {\"status\":\"using already created layer sha256:22f7f8ef5f4c791c1b03d7eb414399294764d7cc82c7e94aa81a1feb80a983a2\"} {\"status\":\"using already created layer sha256:8c17c2ebb0ea011be9981cc3922db8ca8fa61e828c5d3f44cb6ae342bf80460b\"} {\"status\":\"using already created layer sha256:7c23fb36d80141c4ab8cdbb61ee4790102ebd2bf7aeff414453177d4f2110e5d\"} {\"status\":\"using already created layer sha256:2e0493f67d0c8c9c68a8aeacdf6a38a2151cb3c4c1d42accf296e19810527988\"} {\"status\":\"using already created layer sha256:2759286baa875dc22de5394b4a925701b1896a7e3f8e53275c36f75a877a82c9\"} {\"status\":\"writing layer sha256:df30045fe90f0d750db82a058109cecd6d4de9c90a3d75b19c09e5f64580bb42\"} {\"status\":\"writing layer sha256:f18a68eb09bf925bb1b669490407c1b1251c5db98dc4d3d81f3088498ea55690\"} {\"status\":\"writing manifest\"} {\"status\":\"success\"} Check if a Blob Exists HEAD /api/blobs/:digest Ensures that the file blob used for a FROM or ADAPTER field exists on the server. This is checking your Ollama server and not Ollama.ai. Query Parameters digest : the SHA256 digest of the blob Examples Request curl -I http://localhost:11434/api/blobs/sha256:29fdb92e57cf0827ded04ae6461b5931d01fa595843f55d36f5b275a52087dd2 Response Return 200 OK if the blob exists, 404 Not Found if it does not. Create a Blob POST /api/blobs/:digest Create a blob from a file on the server. Returns the server file path. Query Parameters digest : the expected SHA256 digest of the file Examples Request curl -T model.bin -X POST http://localhost:11434/api/blobs/sha256:29fdb92e57cf0827ded04ae6461b5931d01fa595843f55d36f5b275a52087dd2 Response Return 201 Created if the blob was successfully created, 400 Bad Request if the digest used is not expected. List Local Models GET /api/tags List models that are available locally. Examples Request curl http://localhost:11434/api/tags Response A single JSON object will be returned. { \"models\": [ { \"name\": \"codellama:13b\", \"modified_at\": \"2023-11-04T14:56:49.277302595-07:00\", \"size\": 7365960935, \"digest\": \"9f438cb9cd581fc025612d27f7c1a6669ff83a8bb0ed86c94fcf4c5440555697\", \"details\": { \"format\": \"gguf\", \"family\": \"llama\", \"families\": null, \"parameter_size\": \"13B\", \"quantization_level\": \"Q4_0\" } }, { \"name\": \"llama2:latest\", \"modified_at\": \"2023-12-07T09:32:18.757212583-08:00\", \"size\": 3825819519, \"digest\": \"fe938a131f40e6f6d40083c9f0f430a515233eb2edaa6d72eb85c50d64f2300e\", \"details\": { \"format\": \"gguf\", \"family\": \"llama\", \"families\": null, \"parameter_size\": \"7B\", \"quantization_level\": \"Q4_0\" } } ] } Show Model Information POST /api/show Show information about a model including details, modelfile, template, parameters, license, and system prompt. Parameters name : name of the model to show Examples Request curl http://localhost:11434/api/show -d '{ \"name\": \"llama2\" }' Response { \"modelfile\": \"# Modelfile generated by \\\"ollama show\\\"\\n# To build a new Modelfile based on this one, replace the FROM line with:\\n# FROM llava:latest\\n\\nFROM /Users/matt/.ollama/models/blobs/sha256:200765e1283640ffbd013184bf496e261032fa75b99498a9613be4e94d63ad52\\nTEMPLATE \\\"\\\"\\\"{{ .System }}\\nUSER: {{ .Prompt }}\\nASSSISTANT: \\\"\\\"\\\"\\nPARAMETER num_ctx 4096\\nPARAMETER stop \\\"\\u003c/s\\u003e\\\"\\nPARAMETER stop \\\"USER:\\\"\\nPARAMETER stop \\\"ASSSISTANT:\\\"\", \"parameters\": \"num_ctx 4096\\nstop \\u003c/s\\u003e\\nstop USER:\\nstop ASSSISTANT:\", \"template\": \"{{ .System }}\\nUSER: {{ .Prompt }}\\nASSSISTANT: \", \"details\": { \"format\": \"gguf\", \"family\": \"llama\", \"families\": [\"llama\", \"clip\"], \"parameter_size\": \"7B\", \"quantization_level\": \"Q4_0\" } } Copy a Model POST /api/copy Copy a model. Creates a model with another name from an existing model. Examples Request curl http://localhost:11434/api/copy -d '{ \"source\": \"llama2\", \"destination\": \"llama2-backup\" }' Response Returns a 200 OK if successful, or a 404 Not Found if the source model doesn't exist. Delete a Model DELETE /api/delete Delete a model and its data. Parameters name : model name to delete Examples Request curl -X DELETE http://localhost:11434/api/delete -d '{ \"name\": \"llama2:13b\" }' Response Returns a 200 OK if successful, 404 Not Found if the model to be deleted doesn't exist. Pull a Model POST /api/pull Download a model from the ollama library. Cancelled pulls are resumed from where they left off, and multiple calls will share the same download progress. Parameters name : name of the model to pull insecure : (optional) allow insecure connections to the library. Only use this if you are pulling from your own library during development. stream : (optional) if false the response will be returned as a single response object, rather than a stream of objects Examples Request curl http://localhost:11434/api/pull -d '{ \"name\": \"llama2\" }' Response If stream is not specified, or set to true , a stream of JSON objects is returned: The first object is the manifest: { \"status\": \"pulling manifest\" } Then there is a series of downloading responses. Until any of the download is completed, the completed key may not be included. The number of files to be downloaded depends on the number of layers specified in the manifest. { \"status\": \"downloading digestname\", \"digest\": \"digestname\", \"total\": 2142590208, \"completed\": 241970 } After all the files are downloaded, the final responses are: { \"status\": \"verifying sha256 digest\" } { \"status\": \"writing manifest\" } { \"status\": \"removing any unused layers\" } { \"status\": \"success\" } if stream is set to false, then the response is a single JSON object: { \"status\": \"success\" } Push a Model POST /api/push Upload a model to a model library. Requires registering for ollama.ai and adding a public key first. Parameters name : name of the model to push in the form of /: insecure : (optional) allow insecure connections to the library. Only use this if you are pushing to your library during development. stream : (optional) if false the response will be returned as a single response object, rather than a stream of objects Examples Request curl http://localhost:11434/api/push -d '{ \"name\": \"mattw/pygmalion:latest\" }' Response If stream is not specified, or set to true , a stream of JSON objects is returned: { \"status\": \"retrieving manifest\" } and then: { \"status\": \"starting upload\", \"digest\": \"sha256:bc07c81de745696fdf5afca05e065818a8149fb0c77266fb584d9b2cba3711ab\", \"total\": 1928429856 } Then there is a series of uploading responses: { \"status\": \"starting upload\", \"digest\": \"sha256:bc07c81de745696fdf5afca05e065818a8149fb0c77266fb584d9b2cba3711ab\", \"total\": 1928429856 } Finally, when the upload is complete: {\"status\":\"pushing manifest\"} {\"status\":\"success\"} If stream is set to false , then the response is a single JSON object: { \"status\": \"success\" } Generate Embeddings POST /api/embeddings Generate embeddings from a model Parameters model : name of model to generate embeddings from prompt : text to generate embeddings for Advanced parameters: options : additional model parameters listed in the documentation for the Modelfile such as temperature keep_alive : controls how long the model will stay loaded into memory following the request (default: 5m ) Examples Request curl http://localhost:11434/api/embeddings -d '{ \"model\": \"all-minilm\", \"prompt\": \"Here is an article about llamas...\" }' Response { \"embedding\": [ 0.5670403838157654, 0.009260174818336964, 0.23178744316101074, -0.2916173040866852, -0.8924556970596313, 0.8785552978515625, -0.34576427936553955, 0.5742510557174683, -0.04222835972905159, -0.137906014919281 ] }","title":"ollama"},{"location":"ollama/#api","text":"","title":"API"},{"location":"ollama/#endpoints","text":"Generate a completion Generate a chat completion Create a Model List Local Models Show Model Information Copy a Model Delete a Model Pull a Model Push a Model Generate Embeddings","title":"Endpoints"},{"location":"ollama/#conventions","text":"","title":"Conventions"},{"location":"ollama/#model-names","text":"Model names follow a model:tag format, where model can have an optional namespace such as example/model . Some examples are orca-mini:3b-q4_1 and llama2:70b . The tag is optional and, if not provided, will default to latest . The tag is used to identify a specific version.","title":"Model names"},{"location":"ollama/#durations","text":"All durations are returned in nanoseconds.","title":"Durations"},{"location":"ollama/#streaming-responses","text":"Certain endpoints stream responses as JSON objects and can optional return non-streamed responses.","title":"Streaming responses"},{"location":"ollama/#generate-a-completion","text":"POST /api/generate Generate a response for a given prompt with a provided model. This is a streaming endpoint, so there will be a series of responses. The final response object will include statistics and additional data from the request.","title":"Generate a completion"},{"location":"ollama/#parameters","text":"model : (required) the model name prompt : the prompt to generate a response for images : (optional) a list of base64-encoded images (for multimodal models such as llava ) Advanced parameters (optional): format : the format to return a response in. Currently the only accepted value is json options : additional model parameters listed in the documentation for the Modelfile such as temperature system : system message to (overrides what is defined in the Modelfile ) template : the prompt template to use (overrides what is defined in the Modelfile ) context : the context parameter returned from a previous request to /generate , this can be used to keep a short conversational memory stream : if false the response will be returned as a single response object, rather than a stream of objects raw : if true no formatting will be applied to the prompt. You may choose to use the raw parameter if you are specifying a full templated prompt in your request to the API keep_alive : controls how long the model will stay loaded into memory following the request (default: 5m )","title":"Parameters"},{"location":"ollama/#json-mode","text":"Enable JSON mode by setting the format parameter to json . This will structure the response as a valid JSON object. See the JSON mode example below. Note: it's important to instruct the model to use JSON in the prompt . Otherwise, the model may generate large amounts whitespace.","title":"JSON mode"},{"location":"ollama/#examples","text":"","title":"Examples"},{"location":"ollama/#generate-request-streaming","text":"","title":"Generate request (Streaming)"},{"location":"ollama/#request","text":"curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\", \"prompt\": \"Why is the sky blue?\" }'","title":"Request"},{"location":"ollama/#response","text":"A stream of JSON objects is returned: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T08:52:19.385406455-07:00\", \"response\": \"The\", \"done\": false } The final response in the stream also includes additional data about the generation: total_duration : time spent generating the response load_duration : time spent in nanoseconds loading the model prompt_eval_count : number of tokens in the prompt prompt_eval_duration : time spent in nanoseconds evaluating the prompt eval_count : number of tokens the response eval_duration : time in nanoseconds spent generating the response context : an encoding of the conversation used in this response, this can be sent in the next request to keep a conversational memory response : empty if the response was streamed, if not streamed, this will contain the full response To calculate how fast the response is generated in tokens per second (token/s), divide eval_count / eval_duration . { \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"response\": \"\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 10706818083, \"load_duration\": 6338219291, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 130079000, \"eval_count\": 259, \"eval_duration\": 4232710000 }","title":"Response"},{"location":"ollama/#request-no-streaming","text":"","title":"Request (No streaming)"},{"location":"ollama/#request_1","text":"A response can be received in one reply when streaming is off. curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\", \"prompt\": \"Why is the sky blue?\", \"stream\": false }'","title":"Request"},{"location":"ollama/#response_1","text":"If stream is set to false , the response will be a single JSON object: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"response\": \"The sky is blue because it is the color of the sky.\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 5043500667, \"load_duration\": 5025959, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 325953000, \"eval_count\": 290, \"eval_duration\": 4709213000 }","title":"Response"},{"location":"ollama/#request-json-mode","text":"When format is set to json , the output will always be a well-formed JSON object. It's important to also instruct the model to respond in JSON.","title":"Request (JSON mode)"},{"location":"ollama/#request_2","text":"curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\", \"prompt\": \"What color is the sky at different times of the day? Respond using JSON\", \"format\": \"json\", \"stream\": false }'","title":"Request"},{"location":"ollama/#response_2","text":"{ \"model\": \"llama2\", \"created_at\": \"2023-11-09T21:07:55.186497Z\", \"response\": \"{\\n\\\"morning\\\": {\\n\\\"color\\\": \\\"blue\\\"\\n},\\n\\\"noon\\\": {\\n\\\"color\\\": \\\"blue-gray\\\"\\n},\\n\\\"afternoon\\\": {\\n\\\"color\\\": \\\"warm gray\\\"\\n},\\n\\\"evening\\\": {\\n\\\"color\\\": \\\"orange\\\"\\n}\\n}\\n\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 4648158584, \"load_duration\": 4071084, \"prompt_eval_count\": 36, \"prompt_eval_duration\": 439038000, \"eval_count\": 180, \"eval_duration\": 4196918000 } The value of response will be a string containing JSON similar to: { \"morning\": { \"color\": \"blue\" }, \"noon\": { \"color\": \"blue-gray\" }, \"afternoon\": { \"color\": \"warm gray\" }, \"evening\": { \"color\": \"orange\" } }","title":"Response"},{"location":"ollama/#request-with-images","text":"To submit images to multimodal models such as llava or bakllava , provide a list of base64-encoded images :","title":"Request (with images)"},{"location":"ollama/#request_3","text":"curl http://localhost:11434/api/generate -d '{ \"model\": \"llava\", \"prompt\":\"What is in this picture?\", \"stream\": false, \"images\": [\"iVBORw0KGgoAAAANSUhEUgAAAG0AAABmCAYAAADBPx+VAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAA3VSURBVHgB7Z27r0zdG8fX743i1bi1ikMoFMQloXRpKFFIqI7LH4BEQ+NWIkjQuSWCRIEoULk0gsK1kCBI0IhrQVT7tz/7zZo888yz1r7MnDl7z5xvsjkzs2fP3uu71nNfa7lkAsm7d++Sffv2JbNmzUqcc8m0adOSzZs3Z+/XES4ZckAWJEGWPiCxjsQNLWmQsWjRIpMseaxcuTKpG/7HP27I8P79e7dq1ars/yL4/v27S0ejqwv+cUOGEGGpKHR37tzJCEpHV9tnT58+dXXCJDdECBE2Ojrqjh071hpNECjx4cMHVycM1Uhbv359B2F79+51586daxN/+pyRkRFXKyRDAqxEp4yMlDDzXG1NPnnyJKkThoK0VFd1ELZu3TrzXKxKfW7dMBQ6bcuWLW2v0VlHjx41z717927ba22U9APcw7Nnz1oGEPeL3m3p2mTAYYnFmMOMXybPPXv2bNIPpFZr1NHn4HMw0KRBjg9NuRw95s8PEcz/6DZELQd/09C9QGq5RsmSRybqkwHGjh07OsJSsYYm3ijPpyHzoiacg35MLdDSIS/O1yM778jOTwYUkKNHWUzUWaOsylE00MyI0fcnOwIdjvtNdW/HZwNLGg+sR1kMepSNJXmIwxBZiG8tDTpEZzKg0GItNsosY8USkxDhD0Rinuiko2gfL/RbiD2LZAjU9zKQJj8RDR0vJBR1/Phx9+PHj9Z7REF4nTZkxzX4LCXHrV271qXkBAPGfP/atWvu/PnzHe4C97F48eIsRLZ9+3a3f/9+87dwP1JxaF7/3r17ba+5l4EcaVo0lj3SBq5kGTJSQmLWMjgYNei2GPT1MuMqGTDEFHzeQSP2wi/jGnkmPJ/nhccs44jvDAxpVcxnq0F6eT8h4ni/iIWpR5lPyA6ETkNXoSukvpJAD3AsXLiwpZs49+fPn5ke4j10TqYvegSfn0OnafC+Tv9ooA/JPkgQysqQNBzagXY55nO/oa1F7qvIPWkRL12WRpMWUvpVDYmxAPehxWSe8ZEXL20sadYIozfmNch4QJPAfeJgW3rNsnzphBKNJM2KKODo1rVOMRYik5ETy3ix4qWNI81qAAirizgMIc+yhTytx0JWZuNI03qsrgWlGtwjoS9XwgUhWGyhUaRZZQNNIEwCiXD16tXcAHUs79co0vSD8rrJCIW98pzvxpAWyyo3HYwqS0+H0BjStClcZJT5coMm6D2LOF8TolGJtK9fvyZpyiC5ePFi9nc/oJU4eiEP0jVoAnHa9wyJycITMP78+eMeP37sXrx44d6+fdt6f82aNdkx1pg9e3Zb5W+RSRE+n+VjksQWifvVaTKFhn5O8my63K8Qabdv33b379/PiAP//vuvW7BggZszZ072/+TJk91YgkafPn166zXB1rQHFvouAWHq9z3SEevSUerqCn2/dDCeta2jxYbr69evk4MHDyY7d+7MjhMnTiTPnz9Pfv/+nfQT2ggpO2dMF8cghuoM7Ygj5iWCqRlGFml0QC/ftGmTmzt3rmsaKDsgBSPh0/8yPeLLBihLkOKJc0jp8H8vUzcxIA1k6QJ/c78tWEyj5P3o4u9+jywNPdJi5rAH9x0KHcl4Hg570eQp3+vHXGyrmEeigzQsQsjavXt38ujRo44LQuDDhw+TW7duRS1HGgMxhNXHgflaNTOsHyKvHK5Ijo2jbFjJBQK9YwFd6RVMzfgRBmEfP37suBBm/p49e1qjEP2mwTViNRo0VJWH1deMXcNK08uUjVUu7s/zRaL+oLNxz1bpANco4npUgX4G2eFbpDFyQoQxojBCpEGSytmOH8qrH5Q9vuzD6ofQylkCUmh8DBAr+q8JCyVNtWQIidKQE9wNtLSQnS4jDSsxNHogzFuQBw4cyM61UKVsjfr3ooBkPSqqQHesUPWVtzi9/vQi1T+rJj7WiTz4Pt/l3LxUkr5P2VYZaZ4URpsE+st/dujQoaBBYokbrz/8TJNQYLSonrPS9kUaSkPeZyj1AWSj+d+VBoy1pIWVNed8P0Ll/ee5HdGRhrHhR5GGN0r4LGZBaj8oFDJitBTJzIZgFcmU0Y8ytWMZMzJOaXUSrUs5RxKnrxmbb5YXO9VGUhtpXldhEUogFr3IzIsvlpmdosVcGVGXFWp2oU9kLFL3dEkSz6NHEY1sjSRdIuDFWEhd8KxFqsRi1uM/nz9/zpxnwlESONdg6dKlbsaMGS4EHFHtjFIDHwKOo46l4TxSuxgDzi+rE2jg+BaFruOX4HXa0Nnf1lwAPufZeF8/r6zD97WK2qFnGjBxTw5qNGPxT+5T/r7/7RawFC3j4vTp09koCxkeHjqbHJqArmH5UrFKKksnxrK7FuRIs8STfBZv+luugXZ2pR/pP9Ois4z+TiMzUUkUjD0iEi1fzX8GmXyuxUBRcaUfykV0YZnlJGKQpOiGB76x5GeWkWWJc3mOrK6S7xdND+W5N6XyaRgtWJFe13GkaZnKOsYqGdOVVVbGupsyA/l7emTLHi7vwTdirNEt0qxnzAvBFcnQF16xh/TMpUuXHDowhlA9vQVraQhkudRdzOnK+04ZSP3DUhVSP61YsaLtd/ks7ZgtPcXqPqEafHkdqa84X6aCeL7YWlv6edGFHb+ZFICPlljHhg0bKuk0CSvVznWsotRu433alNdFrqG45ejoaPCaUkWERpLXjzFL2Rpllp7PJU2a/v7Ab8N05/9t27Z16KUqoFGsxnI9EosS2niSYg9SpU6B4JgTrvVW1flt1sT+0ADIJU2maXzcUTraGCRaL1Wp9rUMk16PMom8QhruxzvZIegJjFU7LLCePfS8uaQdPny4jTTL0dbee5mYokQsXTIWNY46kuMbnt8Kmec+LGWtOVIl9cT1rCB0V8WqkjAsRwta93TbwNYoGKsUSChN44lgBNCoHLHzquYKrU6qZ8lolCIN0Rh6cP0Q3U6I6IXILYOQI513hJaSKAorFpuHXJNfVlpRtmYBk1Su1obZr5dnKAO+L10Hrj3WZW+E3qh6IszE37F6EB+68mGpvKm4eb9bFrlzrok7fvr0Kfv727dvWRmdVTJHw0qiiCUSZ6wCK+7XL/AcsgNyL74DQQ730sv78Su7+t/A36MdY0sW5o40ahslXr58aZ5HtZB8GH64m9EmMZ7FpYw4T6QnrZfgenrhFxaSiSGXtPnz57e9TkNZLvTjeqhr734CNtrK41L40sUQckmj1lGKQ0rC37x544r8eNXRpnVE3ZZY7zXo8NomiO0ZUCj2uHz58rbXoZ6gc0uA+F6ZeKS/jhRDUq8MKrTho9fEkihMmhxtBI1DxKFY9XLpVcSkfoi8JGnToZO5sU5aiDQIW716ddt7ZLYtMQlhECdBGXZZMWldY5BHm5xgAroWj4C0hbYkSc/jBmggIrXJWlZM6pSETsEPGqZOndr2uuuR5rF169a2HoHPdurUKZM4CO1WTPqaDaAd+GFGKdIQkxAn9RuEWcTRyN2KSUgiSgF5aWzPTeA/lN5rZubMmR2bE4SIC4nJoltgAV/dVefZm72AtctUCJU2CMJ327hxY9t7EHbkyJFseq+EJSY16RPo3Dkq1kkr7+q0bNmyDuLQcZBEPYmHVdOBiJyIlrRDq41YPWfXOxUysi5fvtyaj+2BpcnsUV/oSoEMOk2CQGlr4ckhBwaetBhjCwH0ZHtJROPJkyc7UjcYLDjmrH7ADTEBXFfOYmB0k9oYBOjJ8b4aOYSe7QkKcYhFlq3QYLQhSidNmtS2RATwy8YOM3EQJsUjKiaWZ+vZToUQgzhkHXudb/PW5YMHD9yZM2faPsMwoc7RciYJXbGuBqJ1UIGKKLv915jsvgtJxCZDubdXr165mzdvtr1Hz5LONA8jrUwKPqsmVesKa49S3Q4WxmRPUEYdTjgiUcfUwLx589ySJUva3oMkP6IYddq6HMS4o55xBJBUeRjzfa4Zdeg56QZ43LhxoyPo7Lf1kNt7oO8wWAbNwaYjIv5lhyS7kRf96dvm5Jah8vfvX3flyhX35cuX6HfzFHOToS1H4BenCaHvO8pr8iDuwoUL7tevX+b5ZdbBair0xkFIlFDlW4ZknEClsp/TzXyAKVOmmHWFVSbDNw1l1+4f90U6IY/q4V27dpnE9bJ+v87QEydjqx/UamVVPRG+mwkNTYN+9tjkwzEx+atCm/X9WvWtDtAb68Wy9LXa1UmvCDDIpPkyOQ5ZwSzJ4jMrvFcr0rSjOUh+GcT4LSg5ugkW1Io0/SCDQBojh0hPlaJdah+tkVYrnTZowP8iq1F1TgMBBauufyB33x1v+NWFYmT5KmppgHC+NkAgbmRkpD3yn9QIseXymoTQFGQmIOKTxiZIWpvAatenVqRVXf2nTrAWMsPnKrMZHz6bJq5jvce6QK8J1cQNgKxlJapMPdZSR64/UivS9NztpkVEdKcrs5alhhWP9NeqlfWopzhZScI6QxseegZRGeg5a8C3Re1Mfl1ScP36ddcUaMuv24iOJtz7sbUjTS4qBvKmstYJoUauiuD3k5qhyr7QdUHMeCgLa1Ear9NquemdXgmum4fvJ6w1lqsuDhNrg1qSpleJK7K3TF0Q2jSd94uSZ60kK1e3qyVpQK6PVWXp2/FC3mp6jBhKKOiY2h3gtUV64TWM6wDETRPLDfSakXmH3w8g9Jlug8ZtTt4kVF0kLUYYmCCtD/DrQ5YhMGbA9L3ucdjh0y8kOHW5gU/VEEmJTcL4Pz/f7mgoAbYkAAAAAElFTkSuQmCC\"] }'","title":"Request"},{"location":"ollama/#response_3","text":"{ \"model\": \"llava\", \"created_at\": \"2023-11-03T15:36:02.583064Z\", \"response\": \"A happy cartoon character, which is cute and cheerful.\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 2938432250, \"load_duration\": 2559292, \"prompt_eval_count\": 1, \"prompt_eval_duration\": 2195557000, \"eval_count\": 44, \"eval_duration\": 736432000 }","title":"Response"},{"location":"ollama/#request-raw-mode","text":"In some cases, you may wish to bypass the templating system and provide a full prompt. In this case, you can use the raw parameter to disable templating. Also note that raw mode will not return a context.","title":"Request (Raw Mode)"},{"location":"ollama/#request_4","text":"curl http://localhost:11434/api/generate -d '{ \"model\": \"mistral\", \"prompt\": \"[INST] why is the sky blue? [/INST]\", \"raw\": true, \"stream\": false }'","title":"Request"},{"location":"ollama/#request-reproducible-outputs","text":"For reproducible outputs, set temperature to 0 and seed to a number:","title":"Request (Reproducible outputs)"},{"location":"ollama/#request_5","text":"curl http://localhost:11434/api/generate -d '{ \"model\": \"mistral\", \"prompt\": \"Why is the sky blue?\", \"options\": { \"seed\": 123, \"temperature\": 0 } }'","title":"Request"},{"location":"ollama/#response_4","text":"{ \"model\": \"mistral\", \"created_at\": \"2023-11-03T15:36:02.583064Z\", \"response\": \" The sky appears blue because of a phenomenon called Rayleigh scattering.\", \"done\": true, \"total_duration\": 8493852375, \"load_duration\": 6589624375, \"prompt_eval_count\": 14, \"prompt_eval_duration\": 119039000, \"eval_count\": 110, \"eval_duration\": 1779061000 }","title":"Response"},{"location":"ollama/#generate-request-with-options","text":"If you want to set custom options for the model at runtime rather than in the Modelfile, you can do so with the options parameter. This example sets every available option, but you can set any of them individually and omit the ones you do not want to override.","title":"Generate request (With options)"},{"location":"ollama/#request_6","text":"curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\", \"prompt\": \"Why is the sky blue?\", \"stream\": false, \"options\": { \"num_keep\": 5, \"seed\": 42, \"num_predict\": 100, \"top_k\": 20, \"top_p\": 0.9, \"tfs_z\": 0.5, \"typical_p\": 0.7, \"repeat_last_n\": 33, \"temperature\": 0.8, \"repeat_penalty\": 1.2, \"presence_penalty\": 1.5, \"frequency_penalty\": 1.0, \"mirostat\": 1, \"mirostat_tau\": 0.8, \"mirostat_eta\": 0.6, \"penalize_newline\": true, \"stop\": [\"\\n\", \"user:\"], \"numa\": false, \"num_ctx\": 1024, \"num_batch\": 2, \"num_gqa\": 1, \"num_gpu\": 1, \"main_gpu\": 0, \"low_vram\": false, \"f16_kv\": true, \"vocab_only\": false, \"use_mmap\": true, \"use_mlock\": false, \"rope_frequency_base\": 1.1, \"rope_frequency_scale\": 0.8, \"num_thread\": 8 } }'","title":"Request"},{"location":"ollama/#response_5","text":"{ \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"response\": \"The sky is blue because it is the color of the sky.\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 4935886791, \"load_duration\": 534986708, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 107345000, \"eval_count\": 237, \"eval_duration\": 4289432000 }","title":"Response"},{"location":"ollama/#load-a-model","text":"If an empty prompt is provided, the model will be loaded into memory.","title":"Load a model"},{"location":"ollama/#request_7","text":"curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\" }'","title":"Request"},{"location":"ollama/#response_6","text":"A single JSON object is returned: { \"model\": \"llama2\", \"created_at\": \"2023-12-18T19:52:07.071755Z\", \"response\": \"\", \"done\": true }","title":"Response"},{"location":"ollama/#generate-a-chat-completion","text":"POST /api/chat Generate the next message in a chat with a provided model. This is a streaming endpoint, so there will be a series of responses. Streaming can be disabled using \"stream\": false . The final response object will include statistics and additional data from the request.","title":"Generate a chat completion"},{"location":"ollama/#parameters_1","text":"model : (required) the model name messages : the messages of the chat, this can be used to keep a chat memory The message object has the following fields: role : the role of the message, either system , user or assistant content : the content of the message images (optional): a list of images to include in the message (for multimodal models such as llava ) Advanced parameters (optional): format : the format to return a response in. Currently the only accepted value is json options : additional model parameters listed in the documentation for the Modelfile such as temperature template : the prompt template to use (overrides what is defined in the Modelfile ) stream : if false the response will be returned as a single response object, rather than a stream of objects keep_alive : controls how long the model will stay loaded into memory following the request (default: 5m )","title":"Parameters"},{"location":"ollama/#examples_1","text":"","title":"Examples"},{"location":"ollama/#chat-request-streaming","text":"","title":"Chat Request (Streaming)"},{"location":"ollama/#request_8","text":"Send a chat message with a streaming response. curl http://localhost:11434/api/chat -d '{ \"model\": \"llama2\", \"messages\": [ { \"role\": \"user\", \"content\": \"why is the sky blue?\" } ] }'","title":"Request"},{"location":"ollama/#response_7","text":"A stream of JSON objects is returned: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T08:52:19.385406455-07:00\", \"message\": { \"role\": \"assistant\", \"content\": \"The\", \"images\": null }, \"done\": false } Final response: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"done\": true, \"total_duration\": 4883583458, \"load_duration\": 1334875, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 342546000, \"eval_count\": 282, \"eval_duration\": 4535599000 }","title":"Response"},{"location":"ollama/#chat-request-no-streaming","text":"","title":"Chat request (No streaming)"},{"location":"ollama/#request_9","text":"curl http://localhost:11434/api/chat -d '{ \"model\": \"llama2\", \"messages\": [ { \"role\": \"user\", \"content\": \"why is the sky blue?\" } ], \"stream\": false }'","title":"Request"},{"location":"ollama/#response_8","text":"{ \"model\": \"registry.ollama.ai/library/llama2:latest\", \"created_at\": \"2023-12-12T14:13:43.416799Z\", \"message\": { \"role\": \"assistant\", \"content\": \"Hello! How are you today?\" }, \"done\": true, \"total_duration\": 5191566416, \"load_duration\": 2154458, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 383809000, \"eval_count\": 298, \"eval_duration\": 4799921000 }","title":"Response"},{"location":"ollama/#chat-request-with-history","text":"Send a chat message with a conversation history. You can use this same approach to start the conversation using multi-shot or chain-of-thought prompting.","title":"Chat request (With History)"},{"location":"ollama/#request_10","text":"curl http://localhost:11434/api/chat -d '{ \"model\": \"llama2\", \"messages\": [ { \"role\": \"user\", \"content\": \"why is the sky blue?\" }, { \"role\": \"assistant\", \"content\": \"due to rayleigh scattering.\" }, { \"role\": \"user\", \"content\": \"how is that different than mie scattering?\" } ] }'","title":"Request"},{"location":"ollama/#response_9","text":"A stream of JSON objects is returned: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T08:52:19.385406455-07:00\", \"message\": { \"role\": \"assistant\", \"content\": \"The\" }, \"done\": false } Final response: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"done\": true, \"total_duration\": 8113331500, \"load_duration\": 6396458, \"prompt_eval_count\": 61, \"prompt_eval_duration\": 398801000, \"eval_count\": 468, \"eval_duration\": 7701267000 }","title":"Response"},{"location":"ollama/#chat-request-with-images","text":"","title":"Chat request (with images)"},{"location":"ollama/#request_11","text":"Send a chat message with a conversation history. curl http://localhost:11434/api/chat -d '{ \"model\": \"llava\", \"messages\": [ { \"role\": \"user\", \"content\": \"what is in this image?\", \"images\": [\"iVBORw0KGgoAAAANSUhEUgAAAG0AAABmCAYAAADBPx+VAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAA3VSURBVHgB7Z27r0zdG8fX743i1bi1ikMoFMQloXRpKFFIqI7LH4BEQ+NWIkjQuSWCRIEoULk0gsK1kCBI0IhrQVT7tz/7zZo888yz1r7MnDl7z5xvsjkzs2fP3uu71nNfa7lkAsm7d++Sffv2JbNmzUqcc8m0adOSzZs3Z+/XES4ZckAWJEGWPiCxjsQNLWmQsWjRIpMseaxcuTKpG/7HP27I8P79e7dq1ars/yL4/v27S0ejqwv+cUOGEGGpKHR37tzJCEpHV9tnT58+dXXCJDdECBE2Ojrqjh071hpNECjx4cMHVycM1Uhbv359B2F79+51586daxN/+pyRkRFXKyRDAqxEp4yMlDDzXG1NPnnyJKkThoK0VFd1ELZu3TrzXKxKfW7dMBQ6bcuWLW2v0VlHjx41z717927ba22U9APcw7Nnz1oGEPeL3m3p2mTAYYnFmMOMXybPPXv2bNIPpFZr1NHn4HMw0KRBjg9NuRw95s8PEcz/6DZELQd/09C9QGq5RsmSRybqkwHGjh07OsJSsYYm3ijPpyHzoiacg35MLdDSIS/O1yM778jOTwYUkKNHWUzUWaOsylE00MyI0fcnOwIdjvtNdW/HZwNLGg+sR1kMepSNJXmIwxBZiG8tDTpEZzKg0GItNsosY8USkxDhD0Rinuiko2gfL/RbiD2LZAjU9zKQJj8RDR0vJBR1/Phx9+PHj9Z7REF4nTZkxzX4LCXHrV271qXkBAPGfP/atWvu/PnzHe4C97F48eIsRLZ9+3a3f/9+87dwP1JxaF7/3r17ba+5l4EcaVo0lj3SBq5kGTJSQmLWMjgYNei2GPT1MuMqGTDEFHzeQSP2wi/jGnkmPJ/nhccs44jvDAxpVcxnq0F6eT8h4ni/iIWpR5lPyA6ETkNXoSukvpJAD3AsXLiwpZs49+fPn5ke4j10TqYvegSfn0OnafC+Tv9ooA/JPkgQysqQNBzagXY55nO/oa1F7qvIPWkRL12WRpMWUvpVDYmxAPehxWSe8ZEXL20sadYIozfmNch4QJPAfeJgW3rNsnzphBKNJM2KKODo1rVOMRYik5ETy3ix4qWNI81qAAirizgMIc+yhTytx0JWZuNI03qsrgWlGtwjoS9XwgUhWGyhUaRZZQNNIEwCiXD16tXcAHUs79co0vSD8rrJCIW98pzvxpAWyyo3HYwqS0+H0BjStClcZJT5coMm6D2LOF8TolGJtK9fvyZpyiC5ePFi9nc/oJU4eiEP0jVoAnHa9wyJycITMP78+eMeP37sXrx44d6+fdt6f82aNdkx1pg9e3Zb5W+RSRE+n+VjksQWifvVaTKFhn5O8my63K8Qabdv33b379/PiAP//vuvW7BggZszZ072/+TJk91YgkafPn166zXB1rQHFvouAWHq9z3SEevSUerqCn2/dDCeta2jxYbr69evk4MHDyY7d+7MjhMnTiTPnz9Pfv/+nfQT2ggpO2dMF8cghuoM7Ygj5iWCqRlGFml0QC/ftGmTmzt3rmsaKDsgBSPh0/8yPeLLBihLkOKJc0jp8H8vUzcxIA1k6QJ/c78tWEyj5P3o4u9+jywNPdJi5rAH9x0KHcl4Hg570eQp3+vHXGyrmEeigzQsQsjavXt38ujRo44LQuDDhw+TW7duRS1HGgMxhNXHgflaNTOsHyKvHK5Ijo2jbFjJBQK9YwFd6RVMzfgRBmEfP37suBBm/p49e1qjEP2mwTViNRo0VJWH1deMXcNK08uUjVUu7s/zRaL+oLNxz1bpANco4npUgX4G2eFbpDFyQoQxojBCpEGSytmOH8qrH5Q9vuzD6ofQylkCUmh8DBAr+q8JCyVNtWQIidKQE9wNtLSQnS4jDSsxNHogzFuQBw4cyM61UKVsjfr3ooBkPSqqQHesUPWVtzi9/vQi1T+rJj7WiTz4Pt/l3LxUkr5P2VYZaZ4URpsE+st/dujQoaBBYokbrz/8TJNQYLSonrPS9kUaSkPeZyj1AWSj+d+VBoy1pIWVNed8P0Ll/ee5HdGRhrHhR5GGN0r4LGZBaj8oFDJitBTJzIZgFcmU0Y8ytWMZMzJOaXUSrUs5RxKnrxmbb5YXO9VGUhtpXldhEUogFr3IzIsvlpmdosVcGVGXFWp2oU9kLFL3dEkSz6NHEY1sjSRdIuDFWEhd8KxFqsRi1uM/nz9/zpxnwlESONdg6dKlbsaMGS4EHFHtjFIDHwKOo46l4TxSuxgDzi+rE2jg+BaFruOX4HXa0Nnf1lwAPufZeF8/r6zD97WK2qFnGjBxTw5qNGPxT+5T/r7/7RawFC3j4vTp09koCxkeHjqbHJqArmH5UrFKKksnxrK7FuRIs8STfBZv+luugXZ2pR/pP9Ois4z+TiMzUUkUjD0iEi1fzX8GmXyuxUBRcaUfykV0YZnlJGKQpOiGB76x5GeWkWWJc3mOrK6S7xdND+W5N6XyaRgtWJFe13GkaZnKOsYqGdOVVVbGupsyA/l7emTLHi7vwTdirNEt0qxnzAvBFcnQF16xh/TMpUuXHDowhlA9vQVraQhkudRdzOnK+04ZSP3DUhVSP61YsaLtd/ks7ZgtPcXqPqEafHkdqa84X6aCeL7YWlv6edGFHb+ZFICPlljHhg0bKuk0CSvVznWsotRu433alNdFrqG45ejoaPCaUkWERpLXjzFL2Rpllp7PJU2a/v7Ab8N05/9t27Z16KUqoFGsxnI9EosS2niSYg9SpU6B4JgTrvVW1flt1sT+0ADIJU2maXzcUTraGCRaL1Wp9rUMk16PMom8QhruxzvZIegJjFU7LLCePfS8uaQdPny4jTTL0dbee5mYokQsXTIWNY46kuMbnt8Kmec+LGWtOVIl9cT1rCB0V8WqkjAsRwta93TbwNYoGKsUSChN44lgBNCoHLHzquYKrU6qZ8lolCIN0Rh6cP0Q3U6I6IXILYOQI513hJaSKAorFpuHXJNfVlpRtmYBk1Su1obZr5dnKAO+L10Hrj3WZW+E3qh6IszE37F6EB+68mGpvKm4eb9bFrlzrok7fvr0Kfv727dvWRmdVTJHw0qiiCUSZ6wCK+7XL/AcsgNyL74DQQ730sv78Su7+t/A36MdY0sW5o40ahslXr58aZ5HtZB8GH64m9EmMZ7FpYw4T6QnrZfgenrhFxaSiSGXtPnz57e9TkNZLvTjeqhr734CNtrK41L40sUQckmj1lGKQ0rC37x544r8eNXRpnVE3ZZY7zXo8NomiO0ZUCj2uHz58rbXoZ6gc0uA+F6ZeKS/jhRDUq8MKrTho9fEkihMmhxtBI1DxKFY9XLpVcSkfoi8JGnToZO5sU5aiDQIW716ddt7ZLYtMQlhECdBGXZZMWldY5BHm5xgAroWj4C0hbYkSc/jBmggIrXJWlZM6pSETsEPGqZOndr2uuuR5rF169a2HoHPdurUKZM4CO1WTPqaDaAd+GFGKdIQkxAn9RuEWcTRyN2KSUgiSgF5aWzPTeA/lN5rZubMmR2bE4SIC4nJoltgAV/dVefZm72AtctUCJU2CMJ327hxY9t7EHbkyJFseq+EJSY16RPo3Dkq1kkr7+q0bNmyDuLQcZBEPYmHVdOBiJyIlrRDq41YPWfXOxUysi5fvtyaj+2BpcnsUV/oSoEMOk2CQGlr4ckhBwaetBhjCwH0ZHtJROPJkyc7UjcYLDjmrH7ADTEBXFfOYmB0k9oYBOjJ8b4aOYSe7QkKcYhFlq3QYLQhSidNmtS2RATwy8YOM3EQJsUjKiaWZ+vZToUQgzhkHXudb/PW5YMHD9yZM2faPsMwoc7RciYJXbGuBqJ1UIGKKLv915jsvgtJxCZDubdXr165mzdvtr1Hz5LONA8jrUwKPqsmVesKa49S3Q4WxmRPUEYdTjgiUcfUwLx589ySJUva3oMkP6IYddq6HMS4o55xBJBUeRjzfa4Zdeg56QZ43LhxoyPo7Lf1kNt7oO8wWAbNwaYjIv5lhyS7kRf96dvm5Jah8vfvX3flyhX35cuX6HfzFHOToS1H4BenCaHvO8pr8iDuwoUL7tevX+b5ZdbBair0xkFIlFDlW4ZknEClsp/TzXyAKVOmmHWFVSbDNw1l1+4f90U6IY/q4V27dpnE9bJ+v87QEydjqx/UamVVPRG+mwkNTYN+9tjkwzEx+atCm/X9WvWtDtAb68Wy9LXa1UmvCDDIpPkyOQ5ZwSzJ4jMrvFcr0rSjOUh+GcT4LSg5ugkW1Io0/SCDQBojh0hPlaJdah+tkVYrnTZowP8iq1F1TgMBBauufyB33x1v+NWFYmT5KmppgHC+NkAgbmRkpD3yn9QIseXymoTQFGQmIOKTxiZIWpvAatenVqRVXf2nTrAWMsPnKrMZHz6bJq5jvce6QK8J1cQNgKxlJapMPdZSR64/UivS9NztpkVEdKcrs5alhhWP9NeqlfWopzhZScI6QxseegZRGeg5a8C3Re1Mfl1ScP36ddcUaMuv24iOJtz7sbUjTS4qBvKmstYJoUauiuD3k5qhyr7QdUHMeCgLa1Ear9NquemdXgmum4fvJ6w1lqsuDhNrg1qSpleJK7K3TF0Q2jSd94uSZ60kK1e3qyVpQK6PVWXp2/FC3mp6jBhKKOiY2h3gtUV64TWM6wDETRPLDfSakXmH3w8g9Jlug8ZtTt4kVF0kLUYYmCCtD/DrQ5YhMGbA9L3ucdjh0y8kOHW5gU/VEEmJTcL4Pz/f7mgoAbYkAAAAAElFTkSuQmCC\"] } ] }'","title":"Request"},{"location":"ollama/#response_10","text":"{ \"model\": \"llava\", \"created_at\": \"2023-12-13T22:42:50.203334Z\", \"message\": { \"role\": \"assistant\", \"content\": \" The image features a cute, little pig with an angry facial expression. It's wearing a heart on its shirt and is waving in the air. This scene appears to be part of a drawing or sketching project.\", \"images\": null }, \"done\": true, \"total_duration\": 1668506709, \"load_duration\": 1986209, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 359682000, \"eval_count\": 83, \"eval_duration\": 1303285000 }","title":"Response"},{"location":"ollama/#chat-request-reproducible-outputs","text":"","title":"Chat request (Reproducible outputs)"},{"location":"ollama/#request_12","text":"curl http://localhost:11434/api/chat -d '{ \"model\": \"llama2\", \"messages\": [ { \"role\": \"user\", \"content\": \"Hello!\" } ], \"options\": { \"seed\": 101, \"temperature\": 0 } }'","title":"Request"},{"location":"ollama/#response_11","text":"{ \"model\": \"registry.ollama.ai/library/llama2:latest\", \"created_at\": \"2023-12-12T14:13:43.416799Z\", \"message\": { \"role\": \"assistant\", \"content\": \"Hello! How are you today?\" }, \"done\": true, \"total_duration\": 5191566416, \"load_duration\": 2154458, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 383809000, \"eval_count\": 298, \"eval_duration\": 4799921000 }","title":"Response"},{"location":"ollama/#create-a-model","text":"POST /api/create Create a model from a Modelfile . It is recommended to set modelfile to the content of the Modelfile rather than just set path . This is a requirement for remote create. Remote model creation must also create any file blobs, fields such as FROM and ADAPTER , explicitly with the server using Create a Blob and the value to the path indicated in the response.","title":"Create a Model"},{"location":"ollama/#parameters_2","text":"name : name of the model to create modelfile (optional): contents of the Modelfile stream : (optional) if false the response will be returned as a single response object, rather than a stream of objects path (optional): path to the Modelfile","title":"Parameters"},{"location":"ollama/#examples_2","text":"","title":"Examples"},{"location":"ollama/#create-a-new-model","text":"Create a new model from a Modelfile .","title":"Create a new model"},{"location":"ollama/#request_13","text":"curl http://localhost:11434/api/create -d '{ \"name\": \"mario\", \"modelfile\": \"FROM llama2\\nSYSTEM You are mario from Super Mario Bros.\" }'","title":"Request"},{"location":"ollama/#response_12","text":"A stream of JSON objects. Notice that the final JSON object shows a \"status\": \"success\" . {\"status\":\"reading model metadata\"} {\"status\":\"creating system layer\"} {\"status\":\"using already created layer sha256:22f7f8ef5f4c791c1b03d7eb414399294764d7cc82c7e94aa81a1feb80a983a2\"} {\"status\":\"using already created layer sha256:8c17c2ebb0ea011be9981cc3922db8ca8fa61e828c5d3f44cb6ae342bf80460b\"} {\"status\":\"using already created layer sha256:7c23fb36d80141c4ab8cdbb61ee4790102ebd2bf7aeff414453177d4f2110e5d\"} {\"status\":\"using already created layer sha256:2e0493f67d0c8c9c68a8aeacdf6a38a2151cb3c4c1d42accf296e19810527988\"} {\"status\":\"using already created layer sha256:2759286baa875dc22de5394b4a925701b1896a7e3f8e53275c36f75a877a82c9\"} {\"status\":\"writing layer sha256:df30045fe90f0d750db82a058109cecd6d4de9c90a3d75b19c09e5f64580bb42\"} {\"status\":\"writing layer sha256:f18a68eb09bf925bb1b669490407c1b1251c5db98dc4d3d81f3088498ea55690\"} {\"status\":\"writing manifest\"} {\"status\":\"success\"}","title":"Response"},{"location":"ollama/#check-if-a-blob-exists","text":"HEAD /api/blobs/:digest Ensures that the file blob used for a FROM or ADAPTER field exists on the server. This is checking your Ollama server and not Ollama.ai.","title":"Check if a Blob Exists"},{"location":"ollama/#query-parameters","text":"digest : the SHA256 digest of the blob","title":"Query Parameters"},{"location":"ollama/#examples_3","text":"","title":"Examples"},{"location":"ollama/#request_14","text":"curl -I http://localhost:11434/api/blobs/sha256:29fdb92e57cf0827ded04ae6461b5931d01fa595843f55d36f5b275a52087dd2","title":"Request"},{"location":"ollama/#response_13","text":"Return 200 OK if the blob exists, 404 Not Found if it does not.","title":"Response"},{"location":"ollama/#create-a-blob","text":"POST /api/blobs/:digest Create a blob from a file on the server. Returns the server file path.","title":"Create a Blob"},{"location":"ollama/#query-parameters_1","text":"digest : the expected SHA256 digest of the file","title":"Query Parameters"},{"location":"ollama/#examples_4","text":"","title":"Examples"},{"location":"ollama/#request_15","text":"curl -T model.bin -X POST http://localhost:11434/api/blobs/sha256:29fdb92e57cf0827ded04ae6461b5931d01fa595843f55d36f5b275a52087dd2","title":"Request"},{"location":"ollama/#response_14","text":"Return 201 Created if the blob was successfully created, 400 Bad Request if the digest used is not expected.","title":"Response"},{"location":"ollama/#list-local-models","text":"GET /api/tags List models that are available locally.","title":"List Local Models"},{"location":"ollama/#examples_5","text":"","title":"Examples"},{"location":"ollama/#request_16","text":"curl http://localhost:11434/api/tags","title":"Request"},{"location":"ollama/#response_15","text":"A single JSON object will be returned. { \"models\": [ { \"name\": \"codellama:13b\", \"modified_at\": \"2023-11-04T14:56:49.277302595-07:00\", \"size\": 7365960935, \"digest\": \"9f438cb9cd581fc025612d27f7c1a6669ff83a8bb0ed86c94fcf4c5440555697\", \"details\": { \"format\": \"gguf\", \"family\": \"llama\", \"families\": null, \"parameter_size\": \"13B\", \"quantization_level\": \"Q4_0\" } }, { \"name\": \"llama2:latest\", \"modified_at\": \"2023-12-07T09:32:18.757212583-08:00\", \"size\": 3825819519, \"digest\": \"fe938a131f40e6f6d40083c9f0f430a515233eb2edaa6d72eb85c50d64f2300e\", \"details\": { \"format\": \"gguf\", \"family\": \"llama\", \"families\": null, \"parameter_size\": \"7B\", \"quantization_level\": \"Q4_0\" } } ] }","title":"Response"},{"location":"ollama/#show-model-information","text":"POST /api/show Show information about a model including details, modelfile, template, parameters, license, and system prompt.","title":"Show Model Information"},{"location":"ollama/#parameters_3","text":"name : name of the model to show","title":"Parameters"},{"location":"ollama/#examples_6","text":"","title":"Examples"},{"location":"ollama/#request_17","text":"curl http://localhost:11434/api/show -d '{ \"name\": \"llama2\" }'","title":"Request"},{"location":"ollama/#response_16","text":"{ \"modelfile\": \"# Modelfile generated by \\\"ollama show\\\"\\n# To build a new Modelfile based on this one, replace the FROM line with:\\n# FROM llava:latest\\n\\nFROM /Users/matt/.ollama/models/blobs/sha256:200765e1283640ffbd013184bf496e261032fa75b99498a9613be4e94d63ad52\\nTEMPLATE \\\"\\\"\\\"{{ .System }}\\nUSER: {{ .Prompt }}\\nASSSISTANT: \\\"\\\"\\\"\\nPARAMETER num_ctx 4096\\nPARAMETER stop \\\"\\u003c/s\\u003e\\\"\\nPARAMETER stop \\\"USER:\\\"\\nPARAMETER stop \\\"ASSSISTANT:\\\"\", \"parameters\": \"num_ctx 4096\\nstop \\u003c/s\\u003e\\nstop USER:\\nstop ASSSISTANT:\", \"template\": \"{{ .System }}\\nUSER: {{ .Prompt }}\\nASSSISTANT: \", \"details\": { \"format\": \"gguf\", \"family\": \"llama\", \"families\": [\"llama\", \"clip\"], \"parameter_size\": \"7B\", \"quantization_level\": \"Q4_0\" } }","title":"Response"},{"location":"ollama/#copy-a-model","text":"POST /api/copy Copy a model. Creates a model with another name from an existing model.","title":"Copy a Model"},{"location":"ollama/#examples_7","text":"","title":"Examples"},{"location":"ollama/#request_18","text":"curl http://localhost:11434/api/copy -d '{ \"source\": \"llama2\", \"destination\": \"llama2-backup\" }'","title":"Request"},{"location":"ollama/#response_17","text":"Returns a 200 OK if successful, or a 404 Not Found if the source model doesn't exist.","title":"Response"},{"location":"ollama/#delete-a-model","text":"DELETE /api/delete Delete a model and its data.","title":"Delete a Model"},{"location":"ollama/#parameters_4","text":"name : model name to delete","title":"Parameters"},{"location":"ollama/#examples_8","text":"","title":"Examples"},{"location":"ollama/#request_19","text":"curl -X DELETE http://localhost:11434/api/delete -d '{ \"name\": \"llama2:13b\" }'","title":"Request"},{"location":"ollama/#response_18","text":"Returns a 200 OK if successful, 404 Not Found if the model to be deleted doesn't exist.","title":"Response"},{"location":"ollama/#pull-a-model","text":"POST /api/pull Download a model from the ollama library. Cancelled pulls are resumed from where they left off, and multiple calls will share the same download progress.","title":"Pull a Model"},{"location":"ollama/#parameters_5","text":"name : name of the model to pull insecure : (optional) allow insecure connections to the library. Only use this if you are pulling from your own library during development. stream : (optional) if false the response will be returned as a single response object, rather than a stream of objects","title":"Parameters"},{"location":"ollama/#examples_9","text":"","title":"Examples"},{"location":"ollama/#request_20","text":"curl http://localhost:11434/api/pull -d '{ \"name\": \"llama2\" }'","title":"Request"},{"location":"ollama/#response_19","text":"If stream is not specified, or set to true , a stream of JSON objects is returned: The first object is the manifest: { \"status\": \"pulling manifest\" } Then there is a series of downloading responses. Until any of the download is completed, the completed key may not be included. The number of files to be downloaded depends on the number of layers specified in the manifest. { \"status\": \"downloading digestname\", \"digest\": \"digestname\", \"total\": 2142590208, \"completed\": 241970 } After all the files are downloaded, the final responses are: { \"status\": \"verifying sha256 digest\" } { \"status\": \"writing manifest\" } { \"status\": \"removing any unused layers\" } { \"status\": \"success\" } if stream is set to false, then the response is a single JSON object: { \"status\": \"success\" }","title":"Response"},{"location":"ollama/#push-a-model","text":"POST /api/push Upload a model to a model library. Requires registering for ollama.ai and adding a public key first.","title":"Push a Model"},{"location":"ollama/#parameters_6","text":"name : name of the model to push in the form of /: insecure : (optional) allow insecure connections to the library. Only use this if you are pushing to your library during development. stream : (optional) if false the response will be returned as a single response object, rather than a stream of objects","title":"Parameters"},{"location":"ollama/#examples_10","text":"","title":"Examples"},{"location":"ollama/#request_21","text":"curl http://localhost:11434/api/push -d '{ \"name\": \"mattw/pygmalion:latest\" }'","title":"Request"},{"location":"ollama/#response_20","text":"If stream is not specified, or set to true , a stream of JSON objects is returned: { \"status\": \"retrieving manifest\" } and then: { \"status\": \"starting upload\", \"digest\": \"sha256:bc07c81de745696fdf5afca05e065818a8149fb0c77266fb584d9b2cba3711ab\", \"total\": 1928429856 } Then there is a series of uploading responses: { \"status\": \"starting upload\", \"digest\": \"sha256:bc07c81de745696fdf5afca05e065818a8149fb0c77266fb584d9b2cba3711ab\", \"total\": 1928429856 } Finally, when the upload is complete: {\"status\":\"pushing manifest\"} {\"status\":\"success\"} If stream is set to false , then the response is a single JSON object: { \"status\": \"success\" }","title":"Response"},{"location":"ollama/#generate-embeddings","text":"POST /api/embeddings Generate embeddings from a model","title":"Generate Embeddings"},{"location":"ollama/#parameters_7","text":"model : name of model to generate embeddings from prompt : text to generate embeddings for Advanced parameters: options : additional model parameters listed in the documentation for the Modelfile such as temperature keep_alive : controls how long the model will stay loaded into memory following the request (default: 5m )","title":"Parameters"},{"location":"ollama/#examples_11","text":"","title":"Examples"},{"location":"ollama/#request_22","text":"curl http://localhost:11434/api/embeddings -d '{ \"model\": \"all-minilm\", \"prompt\": \"Here is an article about llamas...\" }'","title":"Request"},{"location":"ollama/#response_21","text":"{ \"embedding\": [ 0.5670403838157654, 0.009260174818336964, 0.23178744316101074, -0.2916173040866852, -0.8924556970596313, 0.8785552978515625, -0.34576427936553955, 0.5742510557174683, -0.04222835972905159, -0.137906014919281 ] }","title":"Response"},{"location":"poetry/","text":"poetry # Install poetry curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python - # Install dependencies poetry install # Run poetry run python main.py # Run tests poetry run pytest # Run tests with coverage and open in browser poetry run pytest --cov=src --cov-report=html && open htmlcov/index.html # Fix poetry shell not activating virtualenv source \"$( poetry env list --full-path | grep Activated | cut -d' ' -f1 )/bin/activate\" # Add from requirements.txt poetry add $(cat requirements.txt) # or cat requirements.txt | xargs poetry add","title":"poetry"},{"location":"poetry/#poetry","text":"# Install poetry curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python - # Install dependencies poetry install # Run poetry run python main.py # Run tests poetry run pytest # Run tests with coverage and open in browser poetry run pytest --cov=src --cov-report=html && open htmlcov/index.html # Fix poetry shell not activating virtualenv source \"$( poetry env list --full-path | grep Activated | cut -d' ' -f1 )/bin/activate\" # Add from requirements.txt poetry add $(cat requirements.txt) # or cat requirements.txt | xargs poetry add","title":"poetry"},{"location":"rag/","text":"RAG Huggingface Retriever used to get documents from vector queries. It retrieves the documents embeddings as well as the documents contents, and it formats them to be used with a RagModel. Examples: # To load the default \"wiki_dpr\" dataset with 21M passages from wikipedia (index name is 'compressed' or 'exact') from transformers import RagRetriever retriever = RagRetriever.from_pretrained( \"facebook/dpr-ctx_encoder-single-nq-base\", dataset=\"wiki_dpr\", index_name=\"compressed\" ) # To load your own indexed dataset built with the datasets library. More info on how to build the indexed dataset in examples/rag/use_own_knowledge_dataset.py from transformers import RagRetriever dataset = ( ... ) # dataset must be a datasets.Datasets object with columns \"title\", \"text\" and \"embeddings\", and it must have a faiss index retriever = RagRetriever.from_pretrained(\"facebook/dpr-ctx_encoder-single-nq-base\", indexed_dataset=dataset) # To load your own indexed dataset built with the datasets library that was saved on disk. More info in examples/rag/use_own_knowledge_dataset.py from transformers import RagRetriever dataset_path = \"path/to/my/dataset\" # dataset saved via *dataset.save_to_disk(...)* index_path = \"path/to/my/index.faiss\" # faiss index saved via *dataset.get_index(\"embeddings\").save(...)* retriever = RagRetriever.from_pretrained( \"facebook/dpr-ctx_encoder-single-nq-base\", index_name=\"custom\", passages_path=dataset_path, index_path=index_path, ) # To load the legacy index built originally for Rag's paper from transformers import RagRetriever retriever = RagRetriever.from_pretrained(\"facebook/dpr-ctx_encoder-single-nq-base\", index_name=\"legacy\") RAG is a seq2seq model which encapsulates two core components: a question encoder and a generator. During a forward pass, we encode the input with the question encoder and pass it to the retriever to extract relevant context documents. The documents are then prepended to the input. Such contextualized inputs is passed to the generator. from transformers import AutoTokenizer, RagRetriever, RagModel import torch tokenizer = AutoTokenizer.from_pretrained(\"facebook/rag-token-base\") retriever = RagRetriever.from_pretrained( \"facebook/rag-token-base\", index_name=\"exact\", use_dummy_dataset=True ) # initialize with RagRetriever to do everything in one forward call model = RagModel.from_pretrained(\"facebook/rag-token-base\", retriever=retriever) inputs = tokenizer(\"How many people live in Paris?\", return_tensors=\"pt\") outputs = model(input_ids=inputs[\"input_ids\"]) from transformers import AutoTokenizer, RagRetriever, RagSequenceForGeneration import torch tokenizer = AutoTokenizer.from_pretrained(\"facebook/rag-sequence-nq\") retriever = RagRetriever.from_pretrained( \"facebook/rag-sequence-nq\", index_name=\"exact\", use_dummy_dataset=True ) # initialize with RagRetriever to do everything in one forward call model = RagSequenceForGeneration.from_pretrained(\"facebook/rag-token-nq\", retriever=retriever) inputs = tokenizer(\"How many people live in Paris?\", return_tensors=\"pt\") targets = tokenizer(text_target=\"In Paris, there are 10 million people.\", return_tensors=\"pt\") input_ids = inputs[\"input_ids\"] labels = targets[\"input_ids\"] outputs = model(input_ids=input_ids, labels=labels) # or use retriever separately model = RagSequenceForGeneration.from_pretrained(\"facebook/rag-sequence-nq\", use_dummy_dataset=True) # 1. Encode question_hidden_states = model.question_encoder(input_ids)[0] # 2. Retrieve docs_dict = retriever(input_ids.numpy(), question_hidden_states.detach().numpy(), return_tensors=\"pt\") doc_scores = torch.bmm( question_hidden_states.unsqueeze(1), docs_dict[\"retrieved_doc_embeds\"].float().transpose(1, 2) ).squeeze(1) # 3. Forward to generator outputs = model( context_input_ids=docs_dict[\"context_input_ids\"], context_attention_mask=docs_dict[\"context_attention_mask\"], doc_scores=doc_scores, decoder_input_ids=labels, ) langchain https://python.langchain.com/docs/expression_language/cookbook/retrieval Let's look at adding in a retrieval step to a prompt and LLM, which adds up to a \"retrieval-augmented generation\" chain pip install langchain openai faiss-cpu tiktoken `````` ```python from operator import itemgetter from langchain.prompts import ChatPromptTemplate from langchain.chat_models import ChatOpenAI from langchain.embeddings import OpenAIEmbeddings from langchain.schema.output_parser import StrOutputParser from langchain.schema.runnable import RunnablePassthrough from langchain.vectorstores import FAISS vectorstore = FAISS.from_texts([\"harrison worked at kensho\"], embedding=OpenAIEmbeddings()) retriever = vectorstore.as_retriever() template = \"\"\"Answer the question based only on the following context: {context} Question: {question} \"\"\" prompt = ChatPromptTemplate.from_template(template) model = ChatOpenAI() chain = ( {\"context\": retriever, \"question\": RunnablePassthrough()} | prompt | model | StrOutputParser() ) chain.invoke(\"where did harrison work?\") template = \"\"\"Answer the question based only on the following context: {context} Question: {question} Answer in the following language: {language} \"\"\" prompt = ChatPromptTemplate.from_template(template) chain = { \"context\": itemgetter(\"question\") | retriever, \"question\": itemgetter(\"question\"), \"language\": itemgetter(\"language\") } | prompt | model | StrOutputParser() chain.invoke({\"question\": \"where did harrison work\", \"language\": \"italian\"})","title":"rag"},{"location":"rag/#rag","text":"","title":"RAG"},{"location":"rag/#huggingface","text":"Retriever used to get documents from vector queries. It retrieves the documents embeddings as well as the documents contents, and it formats them to be used with a RagModel. Examples: # To load the default \"wiki_dpr\" dataset with 21M passages from wikipedia (index name is 'compressed' or 'exact') from transformers import RagRetriever retriever = RagRetriever.from_pretrained( \"facebook/dpr-ctx_encoder-single-nq-base\", dataset=\"wiki_dpr\", index_name=\"compressed\" ) # To load your own indexed dataset built with the datasets library. More info on how to build the indexed dataset in examples/rag/use_own_knowledge_dataset.py from transformers import RagRetriever dataset = ( ... ) # dataset must be a datasets.Datasets object with columns \"title\", \"text\" and \"embeddings\", and it must have a faiss index retriever = RagRetriever.from_pretrained(\"facebook/dpr-ctx_encoder-single-nq-base\", indexed_dataset=dataset) # To load your own indexed dataset built with the datasets library that was saved on disk. More info in examples/rag/use_own_knowledge_dataset.py from transformers import RagRetriever dataset_path = \"path/to/my/dataset\" # dataset saved via *dataset.save_to_disk(...)* index_path = \"path/to/my/index.faiss\" # faiss index saved via *dataset.get_index(\"embeddings\").save(...)* retriever = RagRetriever.from_pretrained( \"facebook/dpr-ctx_encoder-single-nq-base\", index_name=\"custom\", passages_path=dataset_path, index_path=index_path, ) # To load the legacy index built originally for Rag's paper from transformers import RagRetriever retriever = RagRetriever.from_pretrained(\"facebook/dpr-ctx_encoder-single-nq-base\", index_name=\"legacy\") RAG is a seq2seq model which encapsulates two core components: a question encoder and a generator. During a forward pass, we encode the input with the question encoder and pass it to the retriever to extract relevant context documents. The documents are then prepended to the input. Such contextualized inputs is passed to the generator. from transformers import AutoTokenizer, RagRetriever, RagModel import torch tokenizer = AutoTokenizer.from_pretrained(\"facebook/rag-token-base\") retriever = RagRetriever.from_pretrained( \"facebook/rag-token-base\", index_name=\"exact\", use_dummy_dataset=True ) # initialize with RagRetriever to do everything in one forward call model = RagModel.from_pretrained(\"facebook/rag-token-base\", retriever=retriever) inputs = tokenizer(\"How many people live in Paris?\", return_tensors=\"pt\") outputs = model(input_ids=inputs[\"input_ids\"]) from transformers import AutoTokenizer, RagRetriever, RagSequenceForGeneration import torch tokenizer = AutoTokenizer.from_pretrained(\"facebook/rag-sequence-nq\") retriever = RagRetriever.from_pretrained( \"facebook/rag-sequence-nq\", index_name=\"exact\", use_dummy_dataset=True ) # initialize with RagRetriever to do everything in one forward call model = RagSequenceForGeneration.from_pretrained(\"facebook/rag-token-nq\", retriever=retriever) inputs = tokenizer(\"How many people live in Paris?\", return_tensors=\"pt\") targets = tokenizer(text_target=\"In Paris, there are 10 million people.\", return_tensors=\"pt\") input_ids = inputs[\"input_ids\"] labels = targets[\"input_ids\"] outputs = model(input_ids=input_ids, labels=labels) # or use retriever separately model = RagSequenceForGeneration.from_pretrained(\"facebook/rag-sequence-nq\", use_dummy_dataset=True) # 1. Encode question_hidden_states = model.question_encoder(input_ids)[0] # 2. Retrieve docs_dict = retriever(input_ids.numpy(), question_hidden_states.detach().numpy(), return_tensors=\"pt\") doc_scores = torch.bmm( question_hidden_states.unsqueeze(1), docs_dict[\"retrieved_doc_embeds\"].float().transpose(1, 2) ).squeeze(1) # 3. Forward to generator outputs = model( context_input_ids=docs_dict[\"context_input_ids\"], context_attention_mask=docs_dict[\"context_attention_mask\"], doc_scores=doc_scores, decoder_input_ids=labels, )","title":"Huggingface"},{"location":"rag/#langchain","text":"https://python.langchain.com/docs/expression_language/cookbook/retrieval Let's look at adding in a retrieval step to a prompt and LLM, which adds up to a \"retrieval-augmented generation\" chain pip install langchain openai faiss-cpu tiktoken `````` ```python from operator import itemgetter from langchain.prompts import ChatPromptTemplate from langchain.chat_models import ChatOpenAI from langchain.embeddings import OpenAIEmbeddings from langchain.schema.output_parser import StrOutputParser from langchain.schema.runnable import RunnablePassthrough from langchain.vectorstores import FAISS vectorstore = FAISS.from_texts([\"harrison worked at kensho\"], embedding=OpenAIEmbeddings()) retriever = vectorstore.as_retriever() template = \"\"\"Answer the question based only on the following context: {context} Question: {question} \"\"\" prompt = ChatPromptTemplate.from_template(template) model = ChatOpenAI() chain = ( {\"context\": retriever, \"question\": RunnablePassthrough()} | prompt | model | StrOutputParser() ) chain.invoke(\"where did harrison work?\") template = \"\"\"Answer the question based only on the following context: {context} Question: {question} Answer in the following language: {language} \"\"\" prompt = ChatPromptTemplate.from_template(template) chain = { \"context\": itemgetter(\"question\") | retriever, \"question\": itemgetter(\"question\"), \"language\": itemgetter(\"language\") } | prompt | model | StrOutputParser() chain.invoke({\"question\": \"where did harrison work\", \"language\": \"italian\"})","title":"langchain"},{"location":"random/","text":"Random Quartet Hobby became dream. People who cry while eating always survive. https://github.com/aisingapore/PeekingDuck Problem: peeking duck is a opensource modular framework, built for Computer Vision inference. A need for training custom computer vision models with custom datasets. Task: To create a configurable training pipeline for Pytorch and Tensorflow framework for custom datasets for various tasks like classification and object detection. Actions: We used the C4 diagram to blueprint the architecture and selected Hydra as the best configuration framework. The training pipeline was built using the SOLID principles for sustainable development. Results: A highly configurable training pipeline for both Tensorflow and pytorch. Peeking Duck is an open-source modular framework built for Computer Vision inference. The team created a configurable training pipeline for Pytorch and Tensorflow frameworks for custom datasets for various tasks like classification and object detection. They used the C4 diagram to blueprint the architecture and selected Hydra as the best configuration framework. The training pipeline was built using the SOLID principles for sustainable development. The result is a highly configurable training pipeline for both Tensorflow and Pytorch. SOLID is an acronym for five design principles intended to make software designs more understandable, flexible and maintainable. The principles are Single Responsibility Principle (SRP), Open-Closed Principle (OCP), Liskov Substitution Principle (LSP), Interface Segregation Principle (ISP), and Dependency Inversion Principle (DIP).123 SRP: A class should have only one reason to change. OCP: Software entities should be open for extension but closed for modification. LSP: Objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program. ISP: A client should never be forced to implement an interface that it doesn\u2019t use or clients shouldn\u2019t be forced to depend on methods they do not use. DIP: High-level modules should not depend on low-level modules. Both should depend on abstractions. Abstractions should not depend on details. Details should depend on abstractions.","title":"random"},{"location":"random/#random","text":"","title":"Random"},{"location":"random/#quartet","text":"Hobby became dream. People who cry while eating always survive. https://github.com/aisingapore/PeekingDuck Problem: peeking duck is a opensource modular framework, built for Computer Vision inference. A need for training custom computer vision models with custom datasets. Task: To create a configurable training pipeline for Pytorch and Tensorflow framework for custom datasets for various tasks like classification and object detection. Actions: We used the C4 diagram to blueprint the architecture and selected Hydra as the best configuration framework. The training pipeline was built using the SOLID principles for sustainable development. Results: A highly configurable training pipeline for both Tensorflow and pytorch. Peeking Duck is an open-source modular framework built for Computer Vision inference. The team created a configurable training pipeline for Pytorch and Tensorflow frameworks for custom datasets for various tasks like classification and object detection. They used the C4 diagram to blueprint the architecture and selected Hydra as the best configuration framework. The training pipeline was built using the SOLID principles for sustainable development. The result is a highly configurable training pipeline for both Tensorflow and Pytorch. SOLID is an acronym for five design principles intended to make software designs more understandable, flexible and maintainable. The principles are Single Responsibility Principle (SRP), Open-Closed Principle (OCP), Liskov Substitution Principle (LSP), Interface Segregation Principle (ISP), and Dependency Inversion Principle (DIP).123 SRP: A class should have only one reason to change. OCP: Software entities should be open for extension but closed for modification. LSP: Objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program. ISP: A client should never be forced to implement an interface that it doesn\u2019t use or clients shouldn\u2019t be forced to depend on methods they do not use. DIP: High-level modules should not depend on low-level modules. Both should depend on abstractions. Abstractions should not depend on details. Details should depend on abstractions.","title":"Quartet"},{"location":"resume/","text":"Resume \u2584\u2584\u258c \u2590 \u2584\u258c \u2590 \u2584 \u2584\u2584 \u2022 \u2584\u2584\u2584\u2584\u2584\u2584\u2584\u2584 . \u2584\u2584\u00b7 \u2584 \u2022\u2584 \u2022 \u258c \u2584 \u00b7. \u2584\u2584\u2584 . \u2590 \u2584 \u2584\u2584 \u2022 \u2588\u2588\u00b7 \u2588\u258c\u2590\u2588\u25aa \u2022\u2588\u258c\u2590\u2588\u2590\u2588 \u2580 \u25aa \u2022\u2588\u2588 \u2580\u2584.\u2580\u00b7\u2590\u2588 \u258c\u25aa\u2588\u258c\u2584\u258c\u25aa \u00b7\u2588\u2588 \u2590\u2588\u2588\u2588\u25aa\u2580\u2584.\u2580\u00b7\u2022\u2588\u258c\u2590\u2588\u2590\u2588 \u2580 \u25aa \u2588\u2588\u25aa\u2590\u2588\u2590\u2590\u258c \u2584\u2588\u2580\u2584 \u2590\u2588\u2590\u2590\u258c\u2584\u2588 \u2580\u2588\u2584 \u2590\u2588.\u25aa\u2590\u2580\u2580\u25aa\u2584\u2588\u2588 \u2584\u2584\u2590\u2580\u2580\u2584\u00b7 \u2590\u2588 \u258c\u2590\u258c\u2590\u2588\u00b7\u2590\u2580\u2580\u25aa\u2584\u2590\u2588\u2590\u2590\u258c\u2584\u2588 \u2580\u2588\u2584 \u2590\u2588\u258c\u2588\u2588\u2590\u2588\u258c\u2590\u2588\u258c.\u2590\u258c\u2588\u2588\u2590\u2588\u258c\u2590\u2588\u2584\u25aa\u2590\u2588 \u2590\u2588\u258c\u00b7\u2590\u2588\u2584\u2584\u258c\u2590\u2588\u2588\u2588\u258c\u2590\u2588.\u2588\u258c \u2588\u2588 \u2588\u2588\u258c\u2590\u2588\u258c\u2590\u2588\u2584\u2584\u258c\u2588\u2588\u2590\u2588\u258c\u2590\u2588\u2584\u25aa\u2590\u2588 \u2580\u2580\u2580\u2580 \u2580\u25aa \u2580\u2588\u2584\u2580\u25aa\u2580\u2580 \u2588\u25aa\u00b7\u2580\u2580\u2580\u2580 \u2580\u2580\u2580 \u2580\u2580\u2580 \u00b7\u2580\u2580\u2580 \u00b7\u2580 \u2580 \u2580\u2580 \u2588\u25aa\u2580\u2580\u2580 \u2580\u2580\u2580 \u2580\u2580 \u2588\u25aa\u00b7\u2580\u2580\u2580\u2580 20 years of software engineering experience. Reverse engineering codes with strong learning attitude. Particulars Full Name as per passport: Wong Teck Meng Nationality: Singaporean \"Email: \" gfuryhawk@gmail.com Code repositories https://github.com/furyhawk https://www.kaggle.com/teckmengwong Experiences Employment history : 20+ years Software Engineering Period Employer Positions Responsibilities 2023-2024 UParcel AI Engineer Software Engineering 2022-2023 AI Singapore Associate AI Engineer MLOps 2020-2021 NCS Fiori developer SAP Edge Implementation 2019-2020 BHP Developer Fiori Project Implementation 2006-2019 NCS Senior Consultant SAP Consultant/ABAP/Fiori 2000-2006 Creative Associate Engineer Software QA UParcel : 2023 - present AI Engineer Optimize the delivery route for UParcel using large data analysis and parameter fine tuning of heuristic algorithm. Achieved 20% increase in driver acceptance rate of listed jobs. Roll out optimized last mile delivery to Malaysia market. Project management of CD/CI pipeline for UParcel using AWS techstack CDK . AI Singapore : 2022 - 2023 Associate AI Engineer Scoping requirements for content generation of event planning for a government agency using Langchain with prompt engineering . Implement MLOps using Pytorch and Tensorflow framework. Easy configurable( Hydra ) ML pipeline for tasks like Image Classification and Object Detection using SOLID engineering principles. https://github.com/aisingapore/PeekingDuck Research and design industrial defect detection POC for American multinational company using OpenVINO. Deploying to Nvidia Jetson Xavier NX Developer kit. Mentor batch 12 and 13 on Computer Vision. AI Apprentice Out of hundreds applicants, one of the selected 19 in batch 11 . 2 months of deepskilling in AI Engineering and 7 months of on-the-job training on a real-world AI problem from the industry. Deploy scalable distributed container platforms using Kubernetes in GCP . Career transition : 2021 - 2022 : 1 yrs Self-hosted applications and api services on [https://furyhawk.lol/] using Docker Swarm on 3 Raspberry Pi. Transitioning hosted applications and api services to Kubernetes on using Talos and Proxmox . Team Fight Tactics Strategy Application github.com/furyhawk/tftchamp Dockerized application to show gamers meta of current patch using feature importances. Frontend implemented using React and Zustand . NCS : 2020 - 2021 : 1 yrs Fiori developer Implement SAP Edge applications using Agile methodology. Maintenance Dashboard for Singapore Air force F16/Apache Squadron on Edge devices to improve turnaround reliability. BHP : 2019 - 2020 : 1 yr 2 mos Developer Global implementation of Resource Scheduler application that allows for mass scheduling of work to personnel, taking into account work restrictions and capacity availability. Decrease Resource Scheduler Time-on-task , User Error Rate and bolstered safety compliance. NCS : 2008 - 2019 : 10 yrs 7 mos Senior Consultant Successfully implemented 11 cycles of end to end Enterprise Resource Planning ( SAP ) projects, total worth over 100 millions dollars in wide range of business domains(Material Planning, Procurement, Logistics, Finance and Control, Human Resource and Plant Maintenance). Bolstered productivity and add value after each project cycles using Software Engineering best practice . Expertise in implementing projects using Frontend ( Fiori ) and Backend ( ABAP ) frameworks. Spearheaded Agile Scrum methodology to project team members. Restored critical production database outage under time pressure applying root cause analysis techniques . Creative : 2001 - 2008 : 7 yrs 1 mos Associate Engineer Software/Hardware Quality Assurance for world first mp3 devices. Education Qualifications Period Discipline/University 2011-2017 Bachelor of Science IT and Business (ERP) Singapore University of Social Sciences Professional Qualification Certificate Verification \u2013 AI Professionals Association (AIP) Udacity Nanodegree Program - Machine Learning Engineer SMU Certificate in Artificial Intelligence Module 1 - Building AI Capability with Basic Coding for Business SAP Certified Application Associate \u2013 Procurement with SAP ERP 6.0 EHP6 C_FIORDEV_20 Development Associate \u2013 SAP Fiori Application Developer Hobbies - Open source projects","title":"Resume"},{"location":"resume/#resume","text":"\u2584\u2584\u258c \u2590 \u2584\u258c \u2590 \u2584 \u2584\u2584 \u2022 \u2584\u2584\u2584\u2584\u2584\u2584\u2584\u2584 . \u2584\u2584\u00b7 \u2584 \u2022\u2584 \u2022 \u258c \u2584 \u00b7. \u2584\u2584\u2584 . \u2590 \u2584 \u2584\u2584 \u2022 \u2588\u2588\u00b7 \u2588\u258c\u2590\u2588\u25aa \u2022\u2588\u258c\u2590\u2588\u2590\u2588 \u2580 \u25aa \u2022\u2588\u2588 \u2580\u2584.\u2580\u00b7\u2590\u2588 \u258c\u25aa\u2588\u258c\u2584\u258c\u25aa \u00b7\u2588\u2588 \u2590\u2588\u2588\u2588\u25aa\u2580\u2584.\u2580\u00b7\u2022\u2588\u258c\u2590\u2588\u2590\u2588 \u2580 \u25aa \u2588\u2588\u25aa\u2590\u2588\u2590\u2590\u258c \u2584\u2588\u2580\u2584 \u2590\u2588\u2590\u2590\u258c\u2584\u2588 \u2580\u2588\u2584 \u2590\u2588.\u25aa\u2590\u2580\u2580\u25aa\u2584\u2588\u2588 \u2584\u2584\u2590\u2580\u2580\u2584\u00b7 \u2590\u2588 \u258c\u2590\u258c\u2590\u2588\u00b7\u2590\u2580\u2580\u25aa\u2584\u2590\u2588\u2590\u2590\u258c\u2584\u2588 \u2580\u2588\u2584 \u2590\u2588\u258c\u2588\u2588\u2590\u2588\u258c\u2590\u2588\u258c.\u2590\u258c\u2588\u2588\u2590\u2588\u258c\u2590\u2588\u2584\u25aa\u2590\u2588 \u2590\u2588\u258c\u00b7\u2590\u2588\u2584\u2584\u258c\u2590\u2588\u2588\u2588\u258c\u2590\u2588.\u2588\u258c \u2588\u2588 \u2588\u2588\u258c\u2590\u2588\u258c\u2590\u2588\u2584\u2584\u258c\u2588\u2588\u2590\u2588\u258c\u2590\u2588\u2584\u25aa\u2590\u2588 \u2580\u2580\u2580\u2580 \u2580\u25aa \u2580\u2588\u2584\u2580\u25aa\u2580\u2580 \u2588\u25aa\u00b7\u2580\u2580\u2580\u2580 \u2580\u2580\u2580 \u2580\u2580\u2580 \u00b7\u2580\u2580\u2580 \u00b7\u2580 \u2580 \u2580\u2580 \u2588\u25aa\u2580\u2580\u2580 \u2580\u2580\u2580 \u2580\u2580 \u2588\u25aa\u00b7\u2580\u2580\u2580\u2580 20 years of software engineering experience. Reverse engineering codes with strong learning attitude.","title":"Resume"},{"location":"resume/#particulars","text":"Full Name as per passport: Wong Teck Meng Nationality: Singaporean \"Email: \" gfuryhawk@gmail.com Code repositories https://github.com/furyhawk https://www.kaggle.com/teckmengwong","title":"Particulars"},{"location":"resume/#experiences","text":"","title":"Experiences"},{"location":"resume/#employment-history-20-years-software-engineering","text":"Period Employer Positions Responsibilities 2023-2024 UParcel AI Engineer Software Engineering 2022-2023 AI Singapore Associate AI Engineer MLOps 2020-2021 NCS Fiori developer SAP Edge Implementation 2019-2020 BHP Developer Fiori Project Implementation 2006-2019 NCS Senior Consultant SAP Consultant/ABAP/Fiori 2000-2006 Creative Associate Engineer Software QA","title":"Employment history : 20+ years Software Engineering"},{"location":"resume/#uparcel-2023-present","text":"AI Engineer Optimize the delivery route for UParcel using large data analysis and parameter fine tuning of heuristic algorithm. Achieved 20% increase in driver acceptance rate of listed jobs. Roll out optimized last mile delivery to Malaysia market. Project management of CD/CI pipeline for UParcel using AWS techstack CDK .","title":"UParcel : 2023 - present"},{"location":"resume/#ai-singapore-2022-2023","text":"Associate AI Engineer Scoping requirements for content generation of event planning for a government agency using Langchain with prompt engineering . Implement MLOps using Pytorch and Tensorflow framework. Easy configurable( Hydra ) ML pipeline for tasks like Image Classification and Object Detection using SOLID engineering principles. https://github.com/aisingapore/PeekingDuck Research and design industrial defect detection POC for American multinational company using OpenVINO. Deploying to Nvidia Jetson Xavier NX Developer kit. Mentor batch 12 and 13 on Computer Vision. AI Apprentice Out of hundreds applicants, one of the selected 19 in batch 11 . 2 months of deepskilling in AI Engineering and 7 months of on-the-job training on a real-world AI problem from the industry. Deploy scalable distributed container platforms using Kubernetes in GCP .","title":"AI Singapore : 2022 - 2023"},{"location":"resume/#career-transition-2021-2022-1-yrs","text":"Self-hosted applications and api services on [https://furyhawk.lol/] using Docker Swarm on 3 Raspberry Pi. Transitioning hosted applications and api services to Kubernetes on using Talos and Proxmox . Team Fight Tactics Strategy Application github.com/furyhawk/tftchamp Dockerized application to show gamers meta of current patch using feature importances. Frontend implemented using React and Zustand .","title":"Career transition : 2021 - 2022 : 1 yrs"},{"location":"resume/#ncs-2020-2021-1-yrs","text":"Fiori developer Implement SAP Edge applications using Agile methodology. Maintenance Dashboard for Singapore Air force F16/Apache Squadron on Edge devices to improve turnaround reliability.","title":"NCS : 2020 - 2021 : 1 yrs"},{"location":"resume/#bhp-2019-2020-1-yr-2-mos","text":"Developer Global implementation of Resource Scheduler application that allows for mass scheduling of work to personnel, taking into account work restrictions and capacity availability. Decrease Resource Scheduler Time-on-task , User Error Rate and bolstered safety compliance.","title":"BHP : 2019 - 2020 : 1 yr 2 mos"},{"location":"resume/#ncs-2008-2019-10-yrs-7-mos","text":"Senior Consultant Successfully implemented 11 cycles of end to end Enterprise Resource Planning ( SAP ) projects, total worth over 100 millions dollars in wide range of business domains(Material Planning, Procurement, Logistics, Finance and Control, Human Resource and Plant Maintenance). Bolstered productivity and add value after each project cycles using Software Engineering best practice . Expertise in implementing projects using Frontend ( Fiori ) and Backend ( ABAP ) frameworks. Spearheaded Agile Scrum methodology to project team members. Restored critical production database outage under time pressure applying root cause analysis techniques .","title":"NCS : 2008 - 2019 : 10 yrs 7 mos"},{"location":"resume/#creative-2001-2008-7-yrs-1-mos","text":"Associate Engineer Software/Hardware Quality Assurance for world first mp3 devices.","title":"Creative : 2001 - 2008 : 7 yrs 1 mos"},{"location":"resume/#education-qualifications","text":"Period Discipline/University 2011-2017 Bachelor of Science IT and Business (ERP) Singapore University of Social Sciences","title":"Education Qualifications"},{"location":"resume/#professional-qualification","text":"Certificate Verification \u2013 AI Professionals Association (AIP) Udacity Nanodegree Program - Machine Learning Engineer SMU Certificate in Artificial Intelligence Module 1 - Building AI Capability with Basic Coding for Business SAP Certified Application Associate \u2013 Procurement with SAP ERP 6.0 EHP6 C_FIORDEV_20 Development Associate \u2013 SAP Fiori Application Developer","title":"Professional Qualification"},{"location":"resume/#hobbies","text":"- Open source projects","title":"Hobbies"},{"location":"scrape_pad/","text":"majaro \u201croot=PARTUUID=1d7d1003-8f39-41da-9db4-f46a07e06335 rw rootwait audit=0 splash plymouth.ignore-serial-consoles\u201d sway \u201croot=PARTUUID=e9389d07-71a8-44a0-b383-1fbe0c22001a\" \"rw rootwait audit=0 splash plymouth.ignore-serial-consoles\u201d Full Name as per passport: Wong Teck Meng Nationality: Singaporean \"Email: \" gfuryhawk@gmail.com Yolox python tools/train.py -f exps/example/custom/nano.py -d 1 -b 2 -c yolox_nano.pth targets.shape torch.Size([2, 120, 5]) inps.shape torch.Size([2, 3, 416, 416]) question org, dept, position level. what are the expectation of me after 6 months. when do I expectate a reply. weak point briefly. stage of job interview before preparation during connection how are you? Thanks for you for asking, I have being looking forward to meeting you. Smile before exit. tell me about yourself 2-3 stories relevant to jobs. 90-120s. STARS. aspiration. overqualifed immediately fulfill the role, take more roles. ideal job work life balance yrself in 2 yrs growing to fill in others roles. (never about title, promotion) I will really appreciate it if you make the first offer. after follow up the next day on thank you note. misc cp /etc/i3status.conf ~/.config/i3status/config systemctl --user start docker-desktop sudo groupadd docker sudo usermod -aG docker $USER","title":"scrape_pad"},{"location":"scrape_pad/#majaro","text":"\u201croot=PARTUUID=1d7d1003-8f39-41da-9db4-f46a07e06335 rw rootwait audit=0 splash plymouth.ignore-serial-consoles\u201d","title":"majaro"},{"location":"scrape_pad/#sway","text":"\u201croot=PARTUUID=e9389d07-71a8-44a0-b383-1fbe0c22001a\" \"rw rootwait audit=0 splash plymouth.ignore-serial-consoles\u201d Full Name as per passport: Wong Teck Meng Nationality: Singaporean \"Email: \" gfuryhawk@gmail.com","title":"sway"},{"location":"scrape_pad/#yolox","text":"python tools/train.py -f exps/example/custom/nano.py -d 1 -b 2 -c yolox_nano.pth targets.shape torch.Size([2, 120, 5]) inps.shape torch.Size([2, 3, 416, 416])","title":"Yolox"},{"location":"scrape_pad/#question","text":"org, dept, position level. what are the expectation of me after 6 months. when do I expectate a reply. weak point briefly.","title":"question"},{"location":"scrape_pad/#stage-of-job-interview","text":"","title":"stage of job interview"},{"location":"scrape_pad/#before","text":"preparation","title":"before"},{"location":"scrape_pad/#during","text":"connection how are you? Thanks for you for asking, I have being looking forward to meeting you. Smile before exit.","title":"during"},{"location":"scrape_pad/#tell-me-about-yourself","text":"2-3 stories relevant to jobs. 90-120s. STARS. aspiration.","title":"tell me about yourself"},{"location":"scrape_pad/#overqualifed","text":"immediately fulfill the role, take more roles.","title":"overqualifed"},{"location":"scrape_pad/#ideal-job","text":"work life balance","title":"ideal job"},{"location":"scrape_pad/#yrself-in-2-yrs","text":"growing to fill in others roles. (never about title, promotion) I will really appreciate it if you make the first offer.","title":"yrself in 2 yrs"},{"location":"scrape_pad/#after","text":"follow up the next day on thank you note.","title":"after"},{"location":"scrape_pad/#misc","text":"cp /etc/i3status.conf ~/.config/i3status/config systemctl --user start docker-desktop sudo groupadd docker sudo usermod -aG docker $USER","title":"misc"},{"location":"smb/","text":"Samba Samba is the standard Windows interoperability suite of programs for Linux and Unix. Since 1992, Samba has provided secure, stable and fast file and print services for all clients using the SMB/CIFS protocol, such as all versions of DOS and Windows, OS/2, Linux and many others. To share files through Samba, see #Server section; to access files shared through Samba on other machines, please see #Client section. Server cd /mnt/ sudo mkdir media nano /home/user/.credentials sudo chmod 400 /home/user/.credentials sudo mount -t cifs -o rw,vers=3.0,credentials=/home/user/.credentials //coco.local/media/ /mnt/media sudo nano /etc/fstab # smbclient -L hostname -U% # //coco.local/media /mnt/media cifs vers=3.0,credentials=/home/user/.credentials Installation Install the package. Samba is configured in the configuration file, which is extensively documented in . Because the package does not provide this file, one needs to create it before starting . A documented example as in from the Samba git repository may be used to setup . Enabling and starting services To provide basic file sharing through SMB, enable/start . See for details. If you want to make your server accessible via NetBIOS host name, set the desired name in the option in and enable/start . See for details. Make the server discoverable Install the package, then enable/start to make the samba server discoverable with Zeroconf . It should work for most non-Windows file managers (macOS Finder, various GUI-based file managers on Linux & BSD etc.) If is not running, the server will still be accessible, just not discoverable, i.e. it will not show up in file managers, but you can still connect to the server directly by IP or domain. Windows Explorer relies on the WS-Discovery protocol instead; see #Windows 1709 or up does not discover the samba server in Network view . Configure firewall If you are using a firewall , do not forget to open required ports (usually 137-139 + 445). For a complete list, see Samba port usage . UFW Rule A Ufw App Profile for SMB/CIFS is included by default with the default installation of UFW in . Allow Samba by running as root. If you deleted the profile, create/edit and add the following content: [Samba] title=LanManager-like file and printer server for Unix description=The Samba software suite is a collection of programs that implements the SMB/CIFS protocol for unix systems, allowing you to serve files and printers to Windows, NT, OS/2 and DOS clients. This protocol is sometimes also referred to as the LanManager or NetBIOS protocol. ports=137,138/udp|139,445/tcp Then load the profile into UFW run as root. Then finally, allow Samba by running as root. firewalld service To configure firewalld to allow Samba in the home zone, run: # firewall-cmd --permanent --add-service={samba,samba-client,samba-dc} --zone=home The three services listed are: : for sharing files with others. : to browse shares on other machines on the network. : for Samba/Active Directory domain controller . ensures the changes remain after is restarted . Basic configuration User management The following section describes creating a local (tdbsam) database of Samba users. For user authentication and other purposes, Samba can also be bound to an Active Directory domain, can itself serve as an Active Directory domain controller, or can be used with an LDAP server. Adding a user Samba requires a Linux user account - you may use an existing user account or create a new one . Although the user name is shared with Linux system, Samba uses a password separate from that of the Linux user accounts. Replace with the chosen Samba user account: # smbpasswd -a samba_user Depending on the server role , existing File permissions and attributes may need to be altered for the Samba user account. If you want the new user only to be allowed to remotely access the file server shares through Samba, you can restrict other login options\uff1a disabling shell - disabling SSH logons - edit , change option Also see Security for hardening your system. Listing users Samba users can be listed using the command: # pdbedit -L -v Changing user password To change a user password, use : # smbpasswd samba_user Creating an anonymous share 1. Create a Linux user which anonymous Samba users will be mapped to. # useradd guest -s /bin/nologin 2. Add the following to : Anonymous users will now be mapped to the Linux user and have the ability to access any directories defined in , which is configured to be in the example above. Make sure that the Linux user has the proper permissions to access files in . Also, make sure shares have been properly defined as per the Share Definitions section of smb.conf.default . Advanced configuration Enable symlink following Then, restart . Enable server-side copy for macOS clients Server-side copy eliminates the need to transfer data between the server and the client when copying files on the server. This is enabled by default, but it doesn't work with macOS clients. If you have macOS clients, you need to add the following configuration to and then restart . Enable Usershares Usershares is a feature that gives non-root users the capability to add, modify, and delete their own share definitions. See . Create a directory for usershares: Create a user group : Change the owner of the directory to and the group to : Change the permissions of the directory so that users in the group can create files. This command also sets sticky bit , which is important to prevent users from deleting usershares of other users: Set the following parameters in the configuration file: Add the user to the sambashare group. Replace with the name of your user: # gpasswd sambashare -a your_username Restart and services. Log out and log back in. If you want to share paths inside your home directory you must make it accessible for the group others . In the GUI, you can use Thunar or Dolphin - right click on any directory and share it on the network. In the CLI, use one of the following commands, replacing italic sharename , user , ... : # net usershare add sharename abspath [ comment ] [ user :{R|D|F}] [guest_ok={y|n}] # net usershare delete sharename # net usershare list wildcard-sharename # net usershare info wildcard-sharename Set and forcing permissions Permissions may be applied to both the server and shares: See for a full overview of possible permission flags and settings. Restrict protocols for better security Append and in to force usage of a minimum and maximum protocol: See in for an overview of supported protocols. For compatibility with older clients and/or servers, you might need to set or , but please note that this makes you vulnerable to exploits in SMB1 including ransomware attacks. Clients using may need to specify the correct , e.g.: # mount -t cifs // SERVER / sharename /mnt/ mountpoint -o username= username ,password= password ,iocharset= utf8 ,vers= 3.1.1 See for more information. Use native SMB transport encryption Native SMB transport encryption is available in SMB version 3.0 or newer. Clients supporting this type of encryption include Windows 8 and newer, Windows server 2012 and newer, and smbclient of Samba 4.1 and newer. To use native SMB transport encryption by default, set the parameter globally and/or by share. Possible values are , (default value), , or : To configure encryption for on the client side, use the option . See for more information, especially the paragraphs Effects for SMB1 and Effects for SMB2 . Disable printer sharing By default Samba shares printers configured using CUPS . If you do not want printers to be shared, use the following settings: Block certain file extensions on Samba share Samba offers an option to block files with certain patterns, like file extensions. This option can be used to prevent dissemination of viruses or to dissuade users from wasting space with certain files. More information about this option can be found in . Improve throughput The default settings should be sufficient for most users. However setting the 'socket options' correct can improve performance, but getting them wrong can degrade it by just as much. Test the effect before making any large changes. Read the man page before applying any of the options listed below. The following settings should be appended to the section of . Setting a deadtime is useful to stop a server's resources from being exhausted by a large number of inactive connections: deadtime = 30 The usage of sendfile may make more efficient use of the system CPU's and cause Samba to be faster: use sendfile = yes Setting min receivefile size allows zero-copy writes directly from network socket buffers into the filesystem buffer cache (if available). It may improve performance but user testing is recommended: min receivefile size = 16384 Increasing the receive/send buffers size and socket optimize flags might be useful to improve throughput. It is recommended to test each flag separately as it may cause issues on some networks: socket options = IPTOS_LOWDELAY TCP_NODELAY IPTOS_THROUGHPUT SO_RCVBUF=131072 SO_SNDBUF=131072 Enable access for old clients/devices Latest versions of Samba no longer offer older authentication methods and protocols which are still used by some older clients (IP cameras, etc). These devices usually require Samba server to allow NTMLv1 authentication and NT1 version of the protocol, known as CIFS. For these devices to work with latest Samba, you need to add these two configuration parameters into section: server min protocol = NT1 ntlm auth = yes Anonymous/guest access to a share requires just the first parameter. If the old device will access with username and password, you also need the add the second line too. Enable Spotlight searching Spotlight allows supporting clients (e.g. MacOS Finder) to quickly search shared files. Install and start/enable OpenSearch . Install , configure the directories you want to index in , and start/enable for periodic indexing. Edit as described in the Samba wiki to enable Spotlight per share, and restart to apply the changes. Client Install for an -like command line interface. See for commonly used commands. For a lightweight alternative (without support for listing public shares, etc.), install that provides . Depending on the desktop environment , GUI methods may be available. See #File manager configuration for use with a file manager. List public shares The following command lists public shares on a server: $ smbclient -L hostname -U% Alternatively, running will show a tree diagram of all the shares. It uses broadcast queries and is therefore not advisable on a network with a lot of computers, but can be helpful for diagnosing if you have the correct sharename. The () option suppresses the password prompt. NetBIOS/WINS host names Samba clients handle NetBIOS host names automatically by default (the behavior is controlled by the option in ). Other programs (including ) typically use Name Service Switch , which does not handle NetBIOS by default. The package provides a libnss driver to resolve NetBIOS host names. To use it, install it along with the package (which provides the winbindd daemon), start/enable and add to the line in : Now, during host resolving (e.g. when using or just ), winbindd will resolve the host name by sending queries using NetBIOS Name Service (NBNS, also known as WINS) protocol. By default it sends a broadcast query to your local network. If you have a WINS server, you can add to and restart , then winbindd and other Samba clients will send unicast queries to the specified IP. If you want to resolve your local host name (specified in the option in ), start/enable , which will handle incoming queries. You can test WINS resolution with . By default it sends broadcast queries to your local network regardless of the option. Note that WINS resolution requires incoming traffic originating from port 137. Disable NetBIOS/WINS support When not using NetBIOS/WINS host name resolution, it may be preferred to disable this protocol: /etc/samba/smb.conf [global] disable netbios = yes dns proxy = no Finally disable / stop . Manual mounting Mount the share using as . Not all the options listed below are needed or desirable: # mount --mkdir -t cifs // SERVER / sharename /mnt/ mountpoint -o username= username ,password= password ,workgroup= workgroup ,iocharset= utf8 ,uid= username ,gid= group The options uid and gid corresponds to the local (e.g. client) user / user group to have read/write access on the given path. Note : If the uid and gid being used does not match the user of the server, the forceuid and forcegid options may be helpful. However note permissions assigned to a file when forceuid or forcegid are in effect may not reflect the the real (server) permissions. See the File And Directory Ownership And Permissions section in mount.cifs(8) \u00a7\u202fFILE AND DIRECTORY OWNERSHIP AND PERMISSIONS for more information. To mount a Windows share without authentication, use \"username=*\". Warning : Using uid and/or gid as mount options may cause I/O errors, it is recommended to set/check correct File permissions and attributes instead. SERVER \u2014 The server name. sharename \u2014 The shared directory. mountpoint \u2014 The local directory where the share will be mounted. [-o options] \u2014 See for more information. Note : Abstain from using a trailing /. //SERVER/sharename/ will not work. If your mount does not work stable, stutters or freezes, try to enable different SMB protocol version with vers= option. For example, vers=2.0 for Windows Vista mount. If having timeouts on a mounted network share with cifs on a shutdown, see wpa_supplicant#Problem with mounted network shares (cifs) and shutdown. Storing share passwords Storing passwords in a world readable file is not recommended. A safer method is to use a credentials file instead, e.g. inside /etc/samba/credentials : /etc/samba/credentials/share username=myuser password=mypass For the mount command replace username=myuser,password=mypass with credentials=/etc/samba/credentials/share . The credential file should explicitly readable/writeable to root: # chown root:root /etc/samba/credentials # chmod 700 /etc/samba/credentials # chmod 600 /etc/samba/credentials/share Automatic mounting Note : You may need to enable systemd-networkd-wait-online.service or NetworkManager-wait-online.service (depending on your setup) to proper enable booting on start-up. Using NetworkManager and GIO/gvfs NetworkManager can be configured to run a script on network status change. This script uses the gio command so that it mounts the Samba shares automatically, the same way your file manager does, as explained below . The script also safely unmounts the Samba shares before the relevant network connection is disabled by listening for the and events. Make the script is executable after creating it. /etc/NetworkManager/dispatcher.d/30-samba.sh #!/bin/sh # Find the connection UUID with \"nmcli con show\" in terminal. # All NetworkManager connection types are supported: wireless, VPN, wired... WANTED_CON_UUID=\"CHANGE-ME-NOW-9c7eff15-010a-4b1c-a786-9b4efa218ba9\" # The user the share will be mounted under USER=\"yourusername\" # The path that appears in your file manager when you manually mount the share you want SMB_URL=\"smb://servername/share\" # Get runtime user directory. If it does not exist, do nothing and just exit XDG_RUNTIME_DIR=$(loginctl show-user --property=RuntimePath --value \"$USER\") || exit 0 if [ \"$CONNECTION_UUID\" = \"$WANTED_CON_UUID\" ]; then # Script parameter $1: network interface name, not used # Script parameter $2: dispatched event case \"$2\" in \"up\"|\"vpn-up\") su $USER -c \"DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus gio mount $SMB_URL\" ;; \"pre-down\"|\"vpn-pre-down\") su $USER -c \"DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus gio mount -uf $SMB_URL\" ;; esac fi Create a symlink inside to catch the events: # ln -s /etc/NetworkManager/dispatcher.d/30-samba.sh /etc/NetworkManager/dispatcher.d/pre-down.d/30-samba.sh As mount entry This is a simple example of a mount entry that requires authentication: As systemd unit Create a new file inside , e.g. . See for details. path to share path to mount the share share mounting options To use , start the unit and enable it to run on system boot. automount To automatically mount a share (when accessed, like autofs), one may use the following automount unit: Disable / stop the unit, and enable / start to automount the share when the mount path is being accessed. smbnetfs First, check if you can see all the shares you are interested in mounting: $ smbtree -U remote_user If that does not work, find and modify the following line in accordingly: domain master = auto Now restart and . If everything works as expected, install . Then, add the following line to : user_allow_other Now copy the directory to your home directory: $ cp -a /etc/smbnetfs/.smb ~ Then create a link to : $ ln -sf /etc/samba/smb.conf ~/.smb/smb.conf If a username and a password are required to access some of the shared folders, edit to include one or more entries like this: It is also possible to add entries for specific hosts to be mounted by smbnetfs, if necessary. More details can be found in . If you are using the Dolphin or GNOME Files , you may want to add the following to to avoid \"Disk full\" errors as smbnetfs by default will report 0 bytes of free space: When you are done with the configuration, you need to run $ chmod 600 ~/.smb/smbnetfs.* Otherwise, smbnetfs complains about 'insecure config file permissions'. Finally, to mount your Samba network neighbourhood to a directory of your choice, call $ smbnetfs mount_point Daemon The Arch Linux package also maintains an additional system-wide operation mode for smbnetfs. To enable it, you need to make the said modifications in the directory . Then, you can start and/or enable the daemon as usual. The system-wide mount point is at . autofs See Autofs for information on the kernel-based automounter for Linux. File manager configuration GNOME Files, Nemo, Caja, Thunar and PCManFM In order to access samba shares through GNOME Files, Nemo, Caja, Thunar or PCManFM, install the package. Press and enter in the location bar to access your share. The mounted share is likely to be present at or in the filesystem. KDE KDE applications (like Dolphin) has the ability to browse Samba shares built in. Use the path to browse the files. If you want to access files from on non-KDE application, you can install . To use a GUI in the KDE System Settings, you will need to install the package. Other graphical environments There are a number of useful programs, but they may need to have packages created for them. This can be done with the Arch package build system. The good thing about these others is that they do not require a particular environment to be installed to support them, and so they bring along less baggage. LinNeighborhood, RUmba, xffm-samba plugin for Xffm are not available in the official repositories or the AUR. As they are not officially (or even unofficially supported), they may be obsolete and may not work at all. Tips and tricks Discovering network shares If nothing is known about other systems on the local network, and automated tools such as smbnetfs are not available, you can manually probe for Samba shares. First, install the and packages. Use nmap to scan your local network to find systems with TCP port 445 open, which is the port used by the SMB protocol. Note that you may need to use or set a custom ping scan type (e.g. ) because Windows systems are usually firewalled. The first result is another system; the second happens to be the client from where this scan was performed. Now you can connect to there IP addresses directly, but if you want to use NetBIOS host names, you can use to check for NetBIOS names. Note that this will not work if NetBIOS is disabled on the server. Regardless of the output, look for \\<20> , which shows the host with open services. Use to list which services are shared on these systems. You can use NetBIOS host name ( in this example) instead of IP when available. If prompted for a password, pressing enter should still display the list: Remote control of Windows computer Samba offers a set of tools for communication with Windows. These can be handy if access to a Windows computer through remote desktop is not an option, as shown by some examples. Send shutdown command with a comment: $ net rpc shutdown -C \"comment\" -I IPADDRESS -U USERNAME%PASSWORD A forced shutdown instead can be invoked by changing -C with comment to a single -f. For a restart, only add -r, followed by a -C or -f. Stop and start services: $ net rpc service stop SERVICENAME -I IPADDRESS -U USERNAME%PASSWORD To see all possible net rpc command: $ net rpc Troubleshooting Failed to start Samba SMB/CIFS server Possible solutions: Check on syntactic errors with . Set correct permissions for and restart : # chmod 0755 /var/cache/samba/msg Permission issues on SELinux SELinux not allow samba to access user home directories by default, to solve this, run: # setsebool -P samba_enable_home_dirs 1 Similarly, and make Samba has the ability to read or \"read and write\" all files. Permission issues on AppArmor If using a share path located outside of a home or usershares directory, whitelist it in . E.g.: No dialect specified on mount The client is using an unsupported SMB/CIFS version that is required by the server. See #Restrict protocols for better security for more information. Unable to overwrite files, permissions errors Possible solutions: Append the mount option to the entry . Add to the section of the server's . Windows clients keep asking for password even if Samba shares are created with guest permissions Set inside the section of : map to guest = Bad Password If you are still using Samba \\< 4.10.10, use instead of . Windows 7 connectivity problems - mount error(12): cannot allocate memory A known Windows 7 bug that causes \"mount error(12): cannot allocate memory\" on an otherwise perfect cifs share on the Linux end can be fixed by setting a few registry keys on the Windows box as follows: (set to ) (set to ) Alternatively, start Command Prompt in Admin Mode and execute the following: reg add \"HKLM\\SYSTEM\\CurrentControlSet\\Control\\Session Manager\\Memory Management\" /v \"LargeSystemCache\" /t REG_DWORD /d 1 /f reg add \"HKLM\\SYSTEM\\CurrentControlSet\\Services\\LanmanServer\\Parameters\" /v \"Size\" /t REG_DWORD /d 3 /f Do one of the following for the settings to take effect: Restart Windows Restart the Server service via services.msc From the Command Prompt run: 'net stop lanmanserver' and 'net start lanmanserver' - The server may automatically restart after stopping it. Original article . Windows 10 1709 and up connectivity problems - \"Windows cannot access\" 0x80004005 This error affects some machines running Windows 10 version 1709 and later. It is not related to SMB1 being disabled in this version but to the fact that Microsoft disabled insecure logons for guests on this version for some, but not others. To fix, open Group Policy Editor (). Navigate to Computer configuration\\administrative templates\\network\\Lanman Workstation > Enable insecure guest logons and enable it. Alternatively,change the following value in the registry: [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Services\\LanmanWorkstation\\Parameters] \"AllowInsecureGuestAuth\"=dword:1 Error: Failed to retrieve printer list: NT_STATUS_UNSUCCESSFUL If you are a home user and using samba purely for file sharing from a server or NAS, you are probably not interested in sharing printers through it. If so, you can prevent this error from occurring by adding the following lines to your : Restart the samba service, , and then check your logs: # cat /var/log/samba/smbd.log and the error should now no longer be appearing. Sharing a folder fails It means that while you are sharing a folder from Dolphin (file manager) and everything seems ok at first, after restarting Dolphin the share icon is gone from the shared folder, and also some output like this in terminal ( Konsole ) output: \u2018net usershare\u2019 returned error 255: net usershare: usershares are currently disabled To fix it, enable usershare as described in #Enable Usershares . \"Browsing\" network fails with \"Failed to retrieve share list from server\" And you are using a firewall (iptables) because you do not trust your local (school, university, hotel) network. This may be due to the following: When the smbclient is browsing the local network it sends out a broadcast request on udp port 137. The servers on the network then reply to your client but as the source address of this reply is different from the destination address iptables saw when sending the request for the listing out, iptables will not recognize the reply as being \"ESTABLISHED\" or \"RELATED\", and hence the packet is dropped. A possible solution is to add: iptables -t raw -A OUTPUT -p udp -m udp --dport 137 -j CT --helper netbios-ns to your iptables setup. For Uncomplicated Firewall , you need to add to the end of the following line in IPT_MODULES=\"nf_conntrack_ftp nf_nat_ftp nf_conntrack_irc nf_nat_irc\" and then run the following commands as root: echo 1 > /proc/sys/net/netfilter/nf_conntrack_helper ufw allow CIFS ufw reload To make this change persistent across reboots, add the following line at the end of : net.netfilter.nf_conntrack_helper=1 Protocol negotiation failed: NT_STATUS_INVALID_NETWORK_RESPONSE The client probably does not have access to shares. Make sure clients' IP address is in line in . Another problem could be, that the client uses an invalid protocol version. To check this try to connect with the where you specify the maximum protocol version manually: $ smbclient -U -L // -m -W If the command was successful then create a configuration file: Connection to SERVER failed: (Error NT_STATUS_UNSUCCESSFUL) You are probably passing a wrong server name to . To find out the server name, run on the server and look at \"Transient hostname\" line Connection to SERVER failed: (Error NT_STATUS_CONNECTION_REFUSED) Make sure that the server has started. The shared directories should exist and be accessible. Protocol negotiation failed: NT_STATUS_CONNECTION_RESET Probably the server is configured not to accept protocol SMB1. Add option in . Or just pass argument to . Password Error when correct credentials are given (error 1326) Samba 4.5 has NTLMv1 authentication disabled by default. It is recommend to install the latest available upgrades on clients and deny access for unsupported clients. If you still need support for very old clients without NTLMv2 support (e.g. Windows XP), it is possible force enable NTLMv1, although this is not recommend for security reasons: If NTLMv2 clients are unable to authenticate when NTLMv1 has been enabled, create the following file on the client: This change also affects samba shares mounted with mount.cifs . If after upgrade to Samba 4.5 your mount fails, add the sec=ntlmssp option to your mount command, e.g. mount.cifs //server/share /mnt/point -o sec=ntlmssp,... See the man page: ntlmssp - Use NTLMv2 password hashing encapsulated in Raw NTLMSSP message. The default in mainline kernel versions prior to v3.8 was sec=ntlm . In v3.8, the default was changed to sec=ntlmssp . Mapping reserved Windows characters Starting with kernel 3.18, the cifs module uses the \"mapposix\" option by default . When mounting a share using unix extensions and a default Samba configuration, files and directories containing one of the seven reserved Windows characters are listed but cannot be accessed. Possible solutions are: Use the undocumented mount option for cifs # mount.cifs //server/share /mnt/point -o nomapposix Configure Samba to remap (\"SFM\", Services for Mac) style characters to the correct native ones using fruit Manually remap forbidden characters using catia The latter approach (using catia or fruit) has the drawback of filtering files with unprintable characters. Folder shared inside graphical environment is not available to guests This section presupposes: Usershares are configured following previous section A shared folder has been created as a non-root user from GUI Guests access has been set to shared folder during creation Samba service has been restarted at least once since last file modification For clarification purpose only, in the following sub-sections is assumed: Shared folder is located inside user home directory path () Shared folder name is MySharedFiles Guest access is read-only. Windows users will access shared folder content without login prompt Verify correct samba configuration Run the following command from a terminal to test configuration file correctness: $ testparm Verify correct shared folder creation Run the following commands from a terminal: $ cd /var/lib/samba/usershares $ ls If everything is fine, you will notice a file named Read the file contents using the following command: $ cat mysharedfiles The terminal output should display something like this: Verify folder access by guest Run the following command from a terminal. If prompted for a password, just press Enter: $ smbclient -L localhost If everything is fine, MySharedFiles should be displayed under column Run the following command in order to access the shared folder as guest (anonymous login) $ smbclient -N //localhost/MySharedFiles If everything is fine samba client prompt will be displayed: smb: \\> From samba prompt verify guest can list directory contents: smb: \\> ls If the error is displayed, the issue is likely to be with Unix directory permissions. Ensure that your samba user has access to the folder and all parent folders. You can test this by sudoing to the user and attempting to list the mount directory, and all of its parents. Mount error: Host is down This error might be seen when mounting shares of Synology NAS servers. Use the mount option to solve it. Software caused connection abort File managers that utilizes can show the error when writing a file to a share/server. This may be due to the server running SMB/CIFS version 1, which many routers use for USB drive sharing (e.g. Belkin routers). To write to these shares specify the CIFS version with the option . E.g.: This can also happen after updating Samba to version 4.11, which deactivates SMB1 as default, and accessing any Samba share. You can reenable it by adding Connection problem (due to authentification error) Be sure that you do not leave any space characters before your username in Samba client configuration file as follows: The correct format is: Windows 1709 or up does not discover the samba server in Network view With Windows 10 version 1511, support for SMBv1 and thus NetBIOS device discovery was disabled by default. Depending on the actual edition, later versions of Windows starting from version 1709 (\"Fall Creators Update\") do not allow the installation of the SMBv1 client anymore. This causes hosts running Samba not to be listed in the Explorer's \"Network (Neighborhood)\" views. While there is no connectivity problem and Samba will still run fine, users might want to have their Samba hosts to be listed by Windows automatically. implements a Web Service Discovery host daemon. This enables (Samba) hosts, like your local NAS device, to be found by Web Service Discovery Clients like Windows. The default settings should work for most installations, all you need to do is start enable . If the default configuration (advertise itself as the machine hostname in group \"WORKGROUP\") should be all you need in most cases. If you need, you can change configuration options by passing additional arguments to wsdd by adding them in (see the manual page for wsdd for details). does the same thing, but is written in C instead of Python. By default, it will look for the and values in . IOS Files can no longer copy-to Samba share on Arch Linux beginning with IOS 14.5 Beginning with IOS 14.5 attempting to transfer from a device running IOS using the \"Files\" app to a samba share on Arch Linux will result in the error: The operation couldn't be completed Operation canceled To correct this problem, add add the following to the global section of your and restart . Comment optional: ## addition for IOS Files transfer-to server vfs object = fruit streams_xattr See https://apple.stackexchange.com/q/424681 Apple.Stackexchange.com - \"The operation couldn't be completed\"/\"Operation canceled\" error message when saving to a Samba share via Files app. Slow initial connections from certain clients without other performance problems Some SMB clients, such as Solid Explorer for Android, take significantly longer to connect to Samba if they fail to resolve the NetBIOS name. Enabling will greatly speed up initial connections if this is the case. Since this is a bug in the client software, please report such cases to the authors of conflicting software. See also Official website Samba: An Introduction Samba 3.2.x HOWTO and Reference Guide (outdated but still most extensive documentation) Wikipedia Gentoo:Samba/Guide Debian:Samba/ServerSimple KSMBD - A linux kernel server which implements SMB3 protocol in kernel space for sharing files over network. Category:Network sharing Category:Servers","title":"Samba"},{"location":"smb/#samba","text":"Samba is the standard Windows interoperability suite of programs for Linux and Unix. Since 1992, Samba has provided secure, stable and fast file and print services for all clients using the SMB/CIFS protocol, such as all versions of DOS and Windows, OS/2, Linux and many others. To share files through Samba, see #Server section; to access files shared through Samba on other machines, please see #Client section.","title":"Samba"},{"location":"smb/#server","text":"cd /mnt/ sudo mkdir media nano /home/user/.credentials sudo chmod 400 /home/user/.credentials sudo mount -t cifs -o rw,vers=3.0,credentials=/home/user/.credentials //coco.local/media/ /mnt/media sudo nano /etc/fstab # smbclient -L hostname -U% # //coco.local/media /mnt/media cifs vers=3.0,credentials=/home/user/.credentials","title":"Server"},{"location":"smb/#installation","text":"Install the package. Samba is configured in the configuration file, which is extensively documented in . Because the package does not provide this file, one needs to create it before starting . A documented example as in from the Samba git repository may be used to setup .","title":"Installation"},{"location":"smb/#enabling-and-starting-services","text":"To provide basic file sharing through SMB, enable/start . See for details. If you want to make your server accessible via NetBIOS host name, set the desired name in the option in and enable/start . See for details.","title":"Enabling and starting services"},{"location":"smb/#make-the-server-discoverable","text":"Install the package, then enable/start to make the samba server discoverable with Zeroconf . It should work for most non-Windows file managers (macOS Finder, various GUI-based file managers on Linux & BSD etc.) If is not running, the server will still be accessible, just not discoverable, i.e. it will not show up in file managers, but you can still connect to the server directly by IP or domain. Windows Explorer relies on the WS-Discovery protocol instead; see #Windows 1709 or up does not discover the samba server in Network view .","title":"Make the server discoverable"},{"location":"smb/#configure-firewall","text":"If you are using a firewall , do not forget to open required ports (usually 137-139 + 445). For a complete list, see Samba port usage .","title":"Configure firewall"},{"location":"smb/#ufw-rule","text":"A Ufw App Profile for SMB/CIFS is included by default with the default installation of UFW in . Allow Samba by running as root. If you deleted the profile, create/edit and add the following content: [Samba] title=LanManager-like file and printer server for Unix description=The Samba software suite is a collection of programs that implements the SMB/CIFS protocol for unix systems, allowing you to serve files and printers to Windows, NT, OS/2 and DOS clients. This protocol is sometimes also referred to as the LanManager or NetBIOS protocol. ports=137,138/udp|139,445/tcp Then load the profile into UFW run as root. Then finally, allow Samba by running as root.","title":"UFW Rule"},{"location":"smb/#firewalld-service","text":"To configure firewalld to allow Samba in the home zone, run: # firewall-cmd --permanent --add-service={samba,samba-client,samba-dc} --zone=home The three services listed are: : for sharing files with others. : to browse shares on other machines on the network. : for Samba/Active Directory domain controller . ensures the changes remain after is restarted .","title":"firewalld service"},{"location":"smb/#basic-configuration","text":"","title":"Basic configuration"},{"location":"smb/#user-management","text":"The following section describes creating a local (tdbsam) database of Samba users. For user authentication and other purposes, Samba can also be bound to an Active Directory domain, can itself serve as an Active Directory domain controller, or can be used with an LDAP server.","title":"User management"},{"location":"smb/#adding-a-user","text":"Samba requires a Linux user account - you may use an existing user account or create a new one . Although the user name is shared with Linux system, Samba uses a password separate from that of the Linux user accounts. Replace with the chosen Samba user account: # smbpasswd -a samba_user Depending on the server role , existing File permissions and attributes may need to be altered for the Samba user account. If you want the new user only to be allowed to remotely access the file server shares through Samba, you can restrict other login options\uff1a disabling shell - disabling SSH logons - edit , change option Also see Security for hardening your system.","title":"Adding a user"},{"location":"smb/#listing-users","text":"Samba users can be listed using the command: # pdbedit -L -v","title":"Listing users"},{"location":"smb/#changing-user-password","text":"To change a user password, use : # smbpasswd samba_user","title":"Changing user password"},{"location":"smb/#creating-an-anonymous-share","text":"1. Create a Linux user which anonymous Samba users will be mapped to. # useradd guest -s /bin/nologin 2. Add the following to : Anonymous users will now be mapped to the Linux user and have the ability to access any directories defined in , which is configured to be in the example above. Make sure that the Linux user has the proper permissions to access files in . Also, make sure shares have been properly defined as per the Share Definitions section of smb.conf.default .","title":"Creating an anonymous share"},{"location":"smb/#advanced-configuration","text":"","title":"Advanced configuration"},{"location":"smb/#enable-symlink-following","text":"Then, restart .","title":"Enable symlink following"},{"location":"smb/#enable-server-side-copy-for-macos-clients","text":"Server-side copy eliminates the need to transfer data between the server and the client when copying files on the server. This is enabled by default, but it doesn't work with macOS clients. If you have macOS clients, you need to add the following configuration to and then restart .","title":"Enable server-side copy for macOS clients"},{"location":"smb/#enable-usershares","text":"Usershares is a feature that gives non-root users the capability to add, modify, and delete their own share definitions. See . Create a directory for usershares: Create a user group : Change the owner of the directory to and the group to : Change the permissions of the directory so that users in the group can create files. This command also sets sticky bit , which is important to prevent users from deleting usershares of other users: Set the following parameters in the configuration file: Add the user to the sambashare group. Replace with the name of your user: # gpasswd sambashare -a your_username Restart and services. Log out and log back in. If you want to share paths inside your home directory you must make it accessible for the group others . In the GUI, you can use Thunar or Dolphin - right click on any directory and share it on the network. In the CLI, use one of the following commands, replacing italic sharename , user , ... : # net usershare add sharename abspath [ comment ] [ user :{R|D|F}] [guest_ok={y|n}] # net usershare delete sharename # net usershare list wildcard-sharename # net usershare info wildcard-sharename","title":"Enable Usershares"},{"location":"smb/#set-and-forcing-permissions","text":"Permissions may be applied to both the server and shares: See for a full overview of possible permission flags and settings.","title":"Set and forcing permissions"},{"location":"smb/#restrict-protocols-for-better-security","text":"Append and in to force usage of a minimum and maximum protocol: See in for an overview of supported protocols. For compatibility with older clients and/or servers, you might need to set or , but please note that this makes you vulnerable to exploits in SMB1 including ransomware attacks. Clients using may need to specify the correct , e.g.: # mount -t cifs // SERVER / sharename /mnt/ mountpoint -o username= username ,password= password ,iocharset= utf8 ,vers= 3.1.1 See for more information.","title":"Restrict protocols for better security"},{"location":"smb/#use-native-smb-transport-encryption","text":"Native SMB transport encryption is available in SMB version 3.0 or newer. Clients supporting this type of encryption include Windows 8 and newer, Windows server 2012 and newer, and smbclient of Samba 4.1 and newer. To use native SMB transport encryption by default, set the parameter globally and/or by share. Possible values are , (default value), , or : To configure encryption for on the client side, use the option . See for more information, especially the paragraphs Effects for SMB1 and Effects for SMB2 .","title":"Use native SMB transport encryption"},{"location":"smb/#disable-printer-sharing","text":"By default Samba shares printers configured using CUPS . If you do not want printers to be shared, use the following settings:","title":"Disable printer sharing"},{"location":"smb/#block-certain-file-extensions-on-samba-share","text":"Samba offers an option to block files with certain patterns, like file extensions. This option can be used to prevent dissemination of viruses or to dissuade users from wasting space with certain files. More information about this option can be found in .","title":"Block certain file extensions on Samba share"},{"location":"smb/#improve-throughput","text":"The default settings should be sufficient for most users. However setting the 'socket options' correct can improve performance, but getting them wrong can degrade it by just as much. Test the effect before making any large changes. Read the man page before applying any of the options listed below. The following settings should be appended to the section of . Setting a deadtime is useful to stop a server's resources from being exhausted by a large number of inactive connections: deadtime = 30 The usage of sendfile may make more efficient use of the system CPU's and cause Samba to be faster: use sendfile = yes Setting min receivefile size allows zero-copy writes directly from network socket buffers into the filesystem buffer cache (if available). It may improve performance but user testing is recommended: min receivefile size = 16384 Increasing the receive/send buffers size and socket optimize flags might be useful to improve throughput. It is recommended to test each flag separately as it may cause issues on some networks: socket options = IPTOS_LOWDELAY TCP_NODELAY IPTOS_THROUGHPUT SO_RCVBUF=131072 SO_SNDBUF=131072","title":"Improve throughput"},{"location":"smb/#enable-access-for-old-clientsdevices","text":"Latest versions of Samba no longer offer older authentication methods and protocols which are still used by some older clients (IP cameras, etc). These devices usually require Samba server to allow NTMLv1 authentication and NT1 version of the protocol, known as CIFS. For these devices to work with latest Samba, you need to add these two configuration parameters into section: server min protocol = NT1 ntlm auth = yes Anonymous/guest access to a share requires just the first parameter. If the old device will access with username and password, you also need the add the second line too.","title":"Enable access for old clients/devices"},{"location":"smb/#enable-spotlight-searching","text":"Spotlight allows supporting clients (e.g. MacOS Finder) to quickly search shared files. Install and start/enable OpenSearch . Install , configure the directories you want to index in , and start/enable for periodic indexing. Edit as described in the Samba wiki to enable Spotlight per share, and restart to apply the changes.","title":"Enable Spotlight searching"},{"location":"smb/#client","text":"Install for an -like command line interface. See for commonly used commands. For a lightweight alternative (without support for listing public shares, etc.), install that provides . Depending on the desktop environment , GUI methods may be available. See #File manager configuration for use with a file manager.","title":"Client"},{"location":"smb/#list-public-shares","text":"The following command lists public shares on a server: $ smbclient -L hostname -U% Alternatively, running will show a tree diagram of all the shares. It uses broadcast queries and is therefore not advisable on a network with a lot of computers, but can be helpful for diagnosing if you have the correct sharename. The () option suppresses the password prompt.","title":"List public shares"},{"location":"smb/#netbioswins-host-names","text":"Samba clients handle NetBIOS host names automatically by default (the behavior is controlled by the option in ). Other programs (including ) typically use Name Service Switch , which does not handle NetBIOS by default. The package provides a libnss driver to resolve NetBIOS host names. To use it, install it along with the package (which provides the winbindd daemon), start/enable and add to the line in : Now, during host resolving (e.g. when using or just ), winbindd will resolve the host name by sending queries using NetBIOS Name Service (NBNS, also known as WINS) protocol. By default it sends a broadcast query to your local network. If you have a WINS server, you can add to and restart , then winbindd and other Samba clients will send unicast queries to the specified IP. If you want to resolve your local host name (specified in the option in ), start/enable , which will handle incoming queries. You can test WINS resolution with . By default it sends broadcast queries to your local network regardless of the option. Note that WINS resolution requires incoming traffic originating from port 137.","title":"NetBIOS/WINS host names"},{"location":"smb/#disable-netbioswins-support","text":"When not using NetBIOS/WINS host name resolution, it may be preferred to disable this protocol: /etc/samba/smb.conf [global] disable netbios = yes dns proxy = no Finally disable / stop .","title":"Disable NetBIOS/WINS support"},{"location":"smb/#manual-mounting","text":"Mount the share using as . Not all the options listed below are needed or desirable: # mount --mkdir -t cifs // SERVER / sharename /mnt/ mountpoint -o username= username ,password= password ,workgroup= workgroup ,iocharset= utf8 ,uid= username ,gid= group The options uid and gid corresponds to the local (e.g. client) user / user group to have read/write access on the given path. Note : If the uid and gid being used does not match the user of the server, the forceuid and forcegid options may be helpful. However note permissions assigned to a file when forceuid or forcegid are in effect may not reflect the the real (server) permissions. See the File And Directory Ownership And Permissions section in mount.cifs(8) \u00a7\u202fFILE AND DIRECTORY OWNERSHIP AND PERMISSIONS for more information. To mount a Windows share without authentication, use \"username=*\". Warning : Using uid and/or gid as mount options may cause I/O errors, it is recommended to set/check correct File permissions and attributes instead. SERVER \u2014 The server name. sharename \u2014 The shared directory. mountpoint \u2014 The local directory where the share will be mounted. [-o options] \u2014 See for more information. Note : Abstain from using a trailing /. //SERVER/sharename/ will not work. If your mount does not work stable, stutters or freezes, try to enable different SMB protocol version with vers= option. For example, vers=2.0 for Windows Vista mount. If having timeouts on a mounted network share with cifs on a shutdown, see wpa_supplicant#Problem with mounted network shares (cifs) and shutdown.","title":"Manual mounting"},{"location":"smb/#storing-share-passwords","text":"Storing passwords in a world readable file is not recommended. A safer method is to use a credentials file instead, e.g. inside /etc/samba/credentials : /etc/samba/credentials/share username=myuser password=mypass For the mount command replace username=myuser,password=mypass with credentials=/etc/samba/credentials/share . The credential file should explicitly readable/writeable to root: # chown root:root /etc/samba/credentials # chmod 700 /etc/samba/credentials # chmod 600 /etc/samba/credentials/share","title":"Storing share passwords"},{"location":"smb/#automatic-mounting","text":"Note : You may need to enable systemd-networkd-wait-online.service or NetworkManager-wait-online.service (depending on your setup) to proper enable booting on start-up.","title":"Automatic mounting"},{"location":"smb/#using-networkmanager-and-giogvfs","text":"NetworkManager can be configured to run a script on network status change. This script uses the gio command so that it mounts the Samba shares automatically, the same way your file manager does, as explained below . The script also safely unmounts the Samba shares before the relevant network connection is disabled by listening for the and events. Make the script is executable after creating it. /etc/NetworkManager/dispatcher.d/30-samba.sh #!/bin/sh # Find the connection UUID with \"nmcli con show\" in terminal. # All NetworkManager connection types are supported: wireless, VPN, wired... WANTED_CON_UUID=\"CHANGE-ME-NOW-9c7eff15-010a-4b1c-a786-9b4efa218ba9\" # The user the share will be mounted under USER=\"yourusername\" # The path that appears in your file manager when you manually mount the share you want SMB_URL=\"smb://servername/share\" # Get runtime user directory. If it does not exist, do nothing and just exit XDG_RUNTIME_DIR=$(loginctl show-user --property=RuntimePath --value \"$USER\") || exit 0 if [ \"$CONNECTION_UUID\" = \"$WANTED_CON_UUID\" ]; then # Script parameter $1: network interface name, not used # Script parameter $2: dispatched event case \"$2\" in \"up\"|\"vpn-up\") su $USER -c \"DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus gio mount $SMB_URL\" ;; \"pre-down\"|\"vpn-pre-down\") su $USER -c \"DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus gio mount -uf $SMB_URL\" ;; esac fi Create a symlink inside to catch the events: # ln -s /etc/NetworkManager/dispatcher.d/30-samba.sh /etc/NetworkManager/dispatcher.d/pre-down.d/30-samba.sh","title":"Using NetworkManager and GIO/gvfs"},{"location":"smb/#as-mount-entry","text":"This is a simple example of a mount entry that requires authentication:","title":"As mount entry"},{"location":"smb/#as-systemd-unit","text":"Create a new file inside , e.g. . See for details. path to share path to mount the share share mounting options To use , start the unit and enable it to run on system boot.","title":"As systemd unit"},{"location":"smb/#automount","text":"To automatically mount a share (when accessed, like autofs), one may use the following automount unit: Disable / stop the unit, and enable / start to automount the share when the mount path is being accessed.","title":"automount"},{"location":"smb/#smbnetfs","text":"First, check if you can see all the shares you are interested in mounting: $ smbtree -U remote_user If that does not work, find and modify the following line in accordingly: domain master = auto Now restart and . If everything works as expected, install . Then, add the following line to : user_allow_other Now copy the directory to your home directory: $ cp -a /etc/smbnetfs/.smb ~ Then create a link to : $ ln -sf /etc/samba/smb.conf ~/.smb/smb.conf If a username and a password are required to access some of the shared folders, edit to include one or more entries like this: It is also possible to add entries for specific hosts to be mounted by smbnetfs, if necessary. More details can be found in . If you are using the Dolphin or GNOME Files , you may want to add the following to to avoid \"Disk full\" errors as smbnetfs by default will report 0 bytes of free space: When you are done with the configuration, you need to run $ chmod 600 ~/.smb/smbnetfs.* Otherwise, smbnetfs complains about 'insecure config file permissions'. Finally, to mount your Samba network neighbourhood to a directory of your choice, call $ smbnetfs mount_point","title":"smbnetfs"},{"location":"smb/#daemon","text":"The Arch Linux package also maintains an additional system-wide operation mode for smbnetfs. To enable it, you need to make the said modifications in the directory . Then, you can start and/or enable the daemon as usual. The system-wide mount point is at .","title":"Daemon"},{"location":"smb/#autofs","text":"See Autofs for information on the kernel-based automounter for Linux.","title":"autofs"},{"location":"smb/#file-manager-configuration","text":"","title":"File manager configuration"},{"location":"smb/#gnome-files-nemo-caja-thunar-and-pcmanfm","text":"In order to access samba shares through GNOME Files, Nemo, Caja, Thunar or PCManFM, install the package. Press and enter in the location bar to access your share. The mounted share is likely to be present at or in the filesystem.","title":"GNOME Files, Nemo, Caja, Thunar and PCManFM"},{"location":"smb/#kde","text":"KDE applications (like Dolphin) has the ability to browse Samba shares built in. Use the path to browse the files. If you want to access files from on non-KDE application, you can install . To use a GUI in the KDE System Settings, you will need to install the package.","title":"KDE"},{"location":"smb/#other-graphical-environments","text":"There are a number of useful programs, but they may need to have packages created for them. This can be done with the Arch package build system. The good thing about these others is that they do not require a particular environment to be installed to support them, and so they bring along less baggage. LinNeighborhood, RUmba, xffm-samba plugin for Xffm are not available in the official repositories or the AUR. As they are not officially (or even unofficially supported), they may be obsolete and may not work at all.","title":"Other graphical environments"},{"location":"smb/#tips-and-tricks","text":"","title":"Tips and tricks"},{"location":"smb/#discovering-network-shares","text":"If nothing is known about other systems on the local network, and automated tools such as smbnetfs are not available, you can manually probe for Samba shares. First, install the and packages. Use nmap to scan your local network to find systems with TCP port 445 open, which is the port used by the SMB protocol. Note that you may need to use or set a custom ping scan type (e.g. ) because Windows systems are usually firewalled. The first result is another system; the second happens to be the client from where this scan was performed. Now you can connect to there IP addresses directly, but if you want to use NetBIOS host names, you can use to check for NetBIOS names. Note that this will not work if NetBIOS is disabled on the server. Regardless of the output, look for \\<20> , which shows the host with open services. Use to list which services are shared on these systems. You can use NetBIOS host name ( in this example) instead of IP when available. If prompted for a password, pressing enter should still display the list:","title":"Discovering network shares"},{"location":"smb/#remote-control-of-windows-computer","text":"Samba offers a set of tools for communication with Windows. These can be handy if access to a Windows computer through remote desktop is not an option, as shown by some examples. Send shutdown command with a comment: $ net rpc shutdown -C \"comment\" -I IPADDRESS -U USERNAME%PASSWORD A forced shutdown instead can be invoked by changing -C with comment to a single -f. For a restart, only add -r, followed by a -C or -f. Stop and start services: $ net rpc service stop SERVICENAME -I IPADDRESS -U USERNAME%PASSWORD To see all possible net rpc command: $ net rpc","title":"Remote control of Windows computer"},{"location":"smb/#troubleshooting","text":"","title":"Troubleshooting"},{"location":"smb/#failed-to-start-samba-smbcifs-server","text":"Possible solutions: Check on syntactic errors with . Set correct permissions for and restart : # chmod 0755 /var/cache/samba/msg","title":"Failed to start Samba SMB/CIFS server"},{"location":"smb/#permission-issues-on-selinux","text":"SELinux not allow samba to access user home directories by default, to solve this, run: # setsebool -P samba_enable_home_dirs 1 Similarly, and make Samba has the ability to read or \"read and write\" all files.","title":"Permission issues on SELinux"},{"location":"smb/#permission-issues-on-apparmor","text":"If using a share path located outside of a home or usershares directory, whitelist it in . E.g.:","title":"Permission issues on AppArmor"},{"location":"smb/#no-dialect-specified-on-mount","text":"The client is using an unsupported SMB/CIFS version that is required by the server. See #Restrict protocols for better security for more information.","title":"No dialect specified on mount"},{"location":"smb/#unable-to-overwrite-files-permissions-errors","text":"Possible solutions: Append the mount option to the entry . Add to the section of the server's .","title":"Unable to overwrite files, permissions errors"},{"location":"smb/#windows-clients-keep-asking-for-password-even-if-samba-shares-are-created-with-guest-permissions","text":"Set inside the section of : map to guest = Bad Password If you are still using Samba \\< 4.10.10, use instead of .","title":"Windows clients keep asking for password even if Samba shares are created with guest permissions"},{"location":"smb/#windows-7-connectivity-problems-mount-error12-cannot-allocate-memory","text":"A known Windows 7 bug that causes \"mount error(12): cannot allocate memory\" on an otherwise perfect cifs share on the Linux end can be fixed by setting a few registry keys on the Windows box as follows: (set to ) (set to ) Alternatively, start Command Prompt in Admin Mode and execute the following: reg add \"HKLM\\SYSTEM\\CurrentControlSet\\Control\\Session Manager\\Memory Management\" /v \"LargeSystemCache\" /t REG_DWORD /d 1 /f reg add \"HKLM\\SYSTEM\\CurrentControlSet\\Services\\LanmanServer\\Parameters\" /v \"Size\" /t REG_DWORD /d 3 /f Do one of the following for the settings to take effect: Restart Windows Restart the Server service via services.msc From the Command Prompt run: 'net stop lanmanserver' and 'net start lanmanserver' - The server may automatically restart after stopping it. Original article .","title":"Windows 7 connectivity problems - mount error(12): cannot allocate memory"},{"location":"smb/#windows-10-1709-and-up-connectivity-problems-windows-cannot-access-0x80004005","text":"This error affects some machines running Windows 10 version 1709 and later. It is not related to SMB1 being disabled in this version but to the fact that Microsoft disabled insecure logons for guests on this version for some, but not others. To fix, open Group Policy Editor (). Navigate to Computer configuration\\administrative templates\\network\\Lanman Workstation > Enable insecure guest logons and enable it. Alternatively,change the following value in the registry: [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Services\\LanmanWorkstation\\Parameters] \"AllowInsecureGuestAuth\"=dword:1","title":"Windows 10 1709 and up connectivity problems - \"Windows cannot access\" 0x80004005"},{"location":"smb/#error-failed-to-retrieve-printer-list-nt_status_unsuccessful","text":"If you are a home user and using samba purely for file sharing from a server or NAS, you are probably not interested in sharing printers through it. If so, you can prevent this error from occurring by adding the following lines to your : Restart the samba service, , and then check your logs: # cat /var/log/samba/smbd.log and the error should now no longer be appearing.","title":"Error: Failed to retrieve printer list: NT_STATUS_UNSUCCESSFUL"},{"location":"smb/#sharing-a-folder-fails","text":"It means that while you are sharing a folder from Dolphin (file manager) and everything seems ok at first, after restarting Dolphin the share icon is gone from the shared folder, and also some output like this in terminal ( Konsole ) output: \u2018net usershare\u2019 returned error 255: net usershare: usershares are currently disabled To fix it, enable usershare as described in #Enable Usershares .","title":"Sharing a folder fails"},{"location":"smb/#browsing-network-fails-with-failed-to-retrieve-share-list-from-server","text":"And you are using a firewall (iptables) because you do not trust your local (school, university, hotel) network. This may be due to the following: When the smbclient is browsing the local network it sends out a broadcast request on udp port 137. The servers on the network then reply to your client but as the source address of this reply is different from the destination address iptables saw when sending the request for the listing out, iptables will not recognize the reply as being \"ESTABLISHED\" or \"RELATED\", and hence the packet is dropped. A possible solution is to add: iptables -t raw -A OUTPUT -p udp -m udp --dport 137 -j CT --helper netbios-ns to your iptables setup. For Uncomplicated Firewall , you need to add to the end of the following line in IPT_MODULES=\"nf_conntrack_ftp nf_nat_ftp nf_conntrack_irc nf_nat_irc\" and then run the following commands as root: echo 1 > /proc/sys/net/netfilter/nf_conntrack_helper ufw allow CIFS ufw reload To make this change persistent across reboots, add the following line at the end of : net.netfilter.nf_conntrack_helper=1","title":"\"Browsing\" network fails with \"Failed to retrieve share list from server\""},{"location":"smb/#protocol-negotiation-failed-nt_status_invalid_network_response","text":"The client probably does not have access to shares. Make sure clients' IP address is in line in . Another problem could be, that the client uses an invalid protocol version. To check this try to connect with the where you specify the maximum protocol version manually: $ smbclient -U -L // -m -W If the command was successful then create a configuration file:","title":"Protocol negotiation failed: NT_STATUS_INVALID_NETWORK_RESPONSE"},{"location":"smb/#connection-to-server-failed-error-nt_status_unsuccessful","text":"You are probably passing a wrong server name to . To find out the server name, run on the server and look at \"Transient hostname\" line","title":"Connection to SERVER failed: (Error NT_STATUS_UNSUCCESSFUL)"},{"location":"smb/#connection-to-server-failed-error-nt_status_connection_refused","text":"Make sure that the server has started. The shared directories should exist and be accessible.","title":"Connection to SERVER failed: (Error NT_STATUS_CONNECTION_REFUSED)"},{"location":"smb/#protocol-negotiation-failed-nt_status_connection_reset","text":"Probably the server is configured not to accept protocol SMB1. Add option in . Or just pass argument to .","title":"Protocol negotiation failed: NT_STATUS_CONNECTION_RESET"},{"location":"smb/#password-error-when-correct-credentials-are-given-error-1326","text":"Samba 4.5 has NTLMv1 authentication disabled by default. It is recommend to install the latest available upgrades on clients and deny access for unsupported clients. If you still need support for very old clients without NTLMv2 support (e.g. Windows XP), it is possible force enable NTLMv1, although this is not recommend for security reasons: If NTLMv2 clients are unable to authenticate when NTLMv1 has been enabled, create the following file on the client: This change also affects samba shares mounted with mount.cifs . If after upgrade to Samba 4.5 your mount fails, add the sec=ntlmssp option to your mount command, e.g. mount.cifs //server/share /mnt/point -o sec=ntlmssp,... See the man page: ntlmssp - Use NTLMv2 password hashing encapsulated in Raw NTLMSSP message. The default in mainline kernel versions prior to v3.8 was sec=ntlm . In v3.8, the default was changed to sec=ntlmssp .","title":"Password Error when correct credentials are given (error 1326)"},{"location":"smb/#mapping-reserved-windows-characters","text":"Starting with kernel 3.18, the cifs module uses the \"mapposix\" option by default . When mounting a share using unix extensions and a default Samba configuration, files and directories containing one of the seven reserved Windows characters are listed but cannot be accessed. Possible solutions are: Use the undocumented mount option for cifs # mount.cifs //server/share /mnt/point -o nomapposix Configure Samba to remap (\"SFM\", Services for Mac) style characters to the correct native ones using fruit Manually remap forbidden characters using catia The latter approach (using catia or fruit) has the drawback of filtering files with unprintable characters.","title":"Mapping reserved Windows characters"},{"location":"smb/#folder-shared-inside-graphical-environment-is-not-available-to-guests","text":"This section presupposes: Usershares are configured following previous section A shared folder has been created as a non-root user from GUI Guests access has been set to shared folder during creation Samba service has been restarted at least once since last file modification For clarification purpose only, in the following sub-sections is assumed: Shared folder is located inside user home directory path () Shared folder name is MySharedFiles Guest access is read-only. Windows users will access shared folder content without login prompt","title":"Folder shared inside graphical environment is not available to guests"},{"location":"smb/#verify-correct-samba-configuration","text":"Run the following command from a terminal to test configuration file correctness: $ testparm","title":"Verify correct samba configuration"},{"location":"smb/#verify-correct-shared-folder-creation","text":"Run the following commands from a terminal: $ cd /var/lib/samba/usershares $ ls If everything is fine, you will notice a file named Read the file contents using the following command: $ cat mysharedfiles The terminal output should display something like this:","title":"Verify correct shared folder creation"},{"location":"smb/#verify-folder-access-by-guest","text":"Run the following command from a terminal. If prompted for a password, just press Enter: $ smbclient -L localhost If everything is fine, MySharedFiles should be displayed under column Run the following command in order to access the shared folder as guest (anonymous login) $ smbclient -N //localhost/MySharedFiles If everything is fine samba client prompt will be displayed: smb: \\> From samba prompt verify guest can list directory contents: smb: \\> ls If the error is displayed, the issue is likely to be with Unix directory permissions. Ensure that your samba user has access to the folder and all parent folders. You can test this by sudoing to the user and attempting to list the mount directory, and all of its parents.","title":"Verify folder access by guest"},{"location":"smb/#mount-error-host-is-down","text":"This error might be seen when mounting shares of Synology NAS servers. Use the mount option to solve it.","title":"Mount error: Host is down"},{"location":"smb/#software-caused-connection-abort","text":"File managers that utilizes can show the error when writing a file to a share/server. This may be due to the server running SMB/CIFS version 1, which many routers use for USB drive sharing (e.g. Belkin routers). To write to these shares specify the CIFS version with the option . E.g.: This can also happen after updating Samba to version 4.11, which deactivates SMB1 as default, and accessing any Samba share. You can reenable it by adding","title":"Software caused connection abort"},{"location":"smb/#connection-problem-due-to-authentification-error","text":"Be sure that you do not leave any space characters before your username in Samba client configuration file as follows: The correct format is:","title":"Connection problem (due to authentification error)"},{"location":"smb/#windows-1709-or-up-does-not-discover-the-samba-server-in-network-view","text":"With Windows 10 version 1511, support for SMBv1 and thus NetBIOS device discovery was disabled by default. Depending on the actual edition, later versions of Windows starting from version 1709 (\"Fall Creators Update\") do not allow the installation of the SMBv1 client anymore. This causes hosts running Samba not to be listed in the Explorer's \"Network (Neighborhood)\" views. While there is no connectivity problem and Samba will still run fine, users might want to have their Samba hosts to be listed by Windows automatically. implements a Web Service Discovery host daemon. This enables (Samba) hosts, like your local NAS device, to be found by Web Service Discovery Clients like Windows. The default settings should work for most installations, all you need to do is start enable . If the default configuration (advertise itself as the machine hostname in group \"WORKGROUP\") should be all you need in most cases. If you need, you can change configuration options by passing additional arguments to wsdd by adding them in (see the manual page for wsdd for details). does the same thing, but is written in C instead of Python. By default, it will look for the and values in .","title":"Windows 1709 or up does not discover the samba server in Network view"},{"location":"smb/#ios-files-can-no-longer-copy-to-samba-share-on-arch-linux-beginning-with-ios-145","text":"Beginning with IOS 14.5 attempting to transfer from a device running IOS using the \"Files\" app to a samba share on Arch Linux will result in the error: The operation couldn't be completed Operation canceled To correct this problem, add add the following to the global section of your and restart . Comment optional: ## addition for IOS Files transfer-to server vfs object = fruit streams_xattr See https://apple.stackexchange.com/q/424681 Apple.Stackexchange.com - \"The operation couldn't be completed\"/\"Operation canceled\" error message when saving to a Samba share via Files app.","title":"IOS Files can no longer copy-to Samba share on Arch Linux beginning with IOS 14.5"},{"location":"smb/#slow-initial-connections-from-certain-clients-without-other-performance-problems","text":"Some SMB clients, such as Solid Explorer for Android, take significantly longer to connect to Samba if they fail to resolve the NetBIOS name. Enabling will greatly speed up initial connections if this is the case. Since this is a bug in the client software, please report such cases to the authors of conflicting software.","title":"Slow initial connections from certain clients without other performance problems"},{"location":"smb/#see-also","text":"Official website Samba: An Introduction Samba 3.2.x HOWTO and Reference Guide (outdated but still most extensive documentation) Wikipedia Gentoo:Samba/Guide Debian:Samba/ServerSimple KSMBD - A linux kernel server which implements SMB3 protocol in kernel space for sharing files over network. Category:Network sharing Category:Servers","title":"See also"},{"location":"smb2/","text":"Samba This document describes how to mount CIFS shares permanently. The shares might be hosted on a Windows computer/server, or on a Linux/UNIX server running Samba . This document also applies to SMBFS shares, which are similar to CIFS but are deprecated and should be avoided if possible ( link ). (This document does not describe how to host the shares yourself, only how to access shares that are hosted somewhere else. For hosting shares, use Samba .) Prerequisites We're assuming that: - Network connections have been configured properly. - Your local (Ubuntu) username is ubuntuusername . - Share username on Windows computer is msusername . - Share password on Windows computer is mspassword . - The Windows computer's name is servername this can be either an IP address or an assigned name). - The name of the share is sharename . - You want to mount the share in /media/windowsshare . CIFS installation On older systems: Mounting unprotected (guest) network folders First, let's create the mount directory. You will need a separate directory for each mount. Then edit your /etc/fstab file (with root privileges) to add this line: Where - guest indicates you don't need a password to access the share, - uid=1000 makes the Linux user specified by the id the owner of the mounted share, allowing them to rename files, - iocharset=utf8 allows access to files with names in non-English languages. This doesn't work with shares of devices like the Buffalo Tera Station, or Windows machines that export their shares using ISO8895-15. - If there is any space in the server path , you need to replace it by \\040, for example //servername/My\\040Documents` After you add the entry to /etc/fstab type: This will (re)mount all entries listed in /etc/fstab. Mount password protected network folders The quickest way to auto-mounting a password-protected share is to edit /etc/fstab (with root privileges), to add this line: This is not a good idea however: /etc/fstab is readable by everyone and so is your Windows password in it. The way around this is to use a credentials file. This is a file that contains just the username and password. Using a text editor, create a file for your remote servers logon credential: Enter your Windows username and password in the file: Save the file, exit the editor. Change the permissions of the file to prevent unwanted access to your credentials: Then edit your /etc/fstab file (with root privileges) to add this line (replacing the insecure line in the example above, if you added it): Save the file, exit the editor. Finally, test the fstab entry by issuing: If there are no errors, you should test how it works after a reboot. Your remote share should mount automatically. Special permissions If you need special permission (like chmod etc.), you'll need to add a uid (short for 'user id') or gid (for 'group id') parameter to the share's mount options. Mount password protected shares using libpam_mount (Ubuntu 9.04) In addition to the initial assumptions, we're assuming that - Your username and password are the same on the Ubuntu machine and on the network drive. Install libpam-mount: Edit /etc/security/pam_mount.conf.xml using your preferred text editor. First, we're moving the user specific config bits to a file which users can actually edit themselves: remove the commenting tags ( ) surrounding the section called . Save the file when done. With this in place, users can create their own \\~/.pam_mount.conf.xml. Add the following: Troubleshooting Login errors If you get the error \"mount error(13) permission denied\", then the server denied your access. Here are the first things to check: - Are you using a valid username and password? Does that account really have access to this folder? - Do you have whitespace in your credentials file? It should be , not . - Do you need a domain? For example, if you are told that your username is , then actually your username is and your domain is . The fstab entry should read: Or: - The security and version settings are interrelated. SMB1 is insecure and no longer supported by default. At first, try to not specify either security or version: do not specify or . If you still have authentication errors then you may need to specify either or or both. You can try the options listed at the [ mount.cifs man page ](http://manpages.ubuntu.com/manpages/raring/en/man8/mount.cifs.8.html \"wikilink\") . The man page list leaves out the option for some reason, but you should try that one as well (`[`see `discussion ](https://bugs.launchpad.net/ubuntu/+source/cifs-utils/+bug/1113395 \"wikilink\") ). Unprotected network folder won't automount I've had a situation where an unprotected network folder wouldn't automount during bootup, but after manually entering \"sudo mount -a\" was mounted correctly. I solved this by replacing the \"guest\" option by \"username=guest,password=\". If anyone has an explanation for this, please leave a comment. Mount during login instead of boot If for some reason/etc/rc0.d/S31umountnfs.sh (networking problems for example) the automatic mounting during boot doesn't work, you can add the \"noauto\" parameter to your smbfs fstab entry and then have the share mounted at login. In /etc/fstab: In /etc/rc.local: Slow shutdown due to a CIFS/Network Manager bug If you use Network Manager, and are getting really slow shutdowns, it's probably because NM shuts down before unmounting the network shares. That will cause CIFS to hang and wait for 60 seconds or so. Here's how to fix it:/etc/rc0.d/S31umountnfs.sh Ubuntu 12.04 already runs umountnfs.sh at reboot and shutdown by default (/etc/rc0.d/S31umountnfs.sh and /etc/rc6.d/S31umountnfs.sh) so this is no longer necessary. CIFS Options Deprecated 20 Feb 2008 TW Using dmask or fmask in the fstab file produces the following warnings: WARNING: CIFS mount option 'dmask' is deprecated. Use 'dir_mode' instead. WARNING: CIFS mount option 'fmask' is deprecated. Use 'file_mode' instead. Instead use this format: file_mode=0777,dir_mode=0777 . Or in some cases you might need to use file_mode=0777,dir_mode=0777,nounix ( see discussion ) \\== Use of tilde in pathnames such as \"credentials=\\~/.smbcredentials\" 20 Feb 2008 TW Curiously, using credentials=\\~/.smbcredentials in fstab didn't work. I had to use the full path, i.e. /home/username/.smbcredentials (This is likely because the tilde \"\\~\" is only a shell short-hand alias for \"$HOME\"; it isn't something recognized system-wide by all programs, especially not in a system file table where the concept of \"HOME\" doesn't really exist. -Ian!) CategoryDocumentation","title":"Samba_ubuntu"},{"location":"smb2/#samba","text":"This document describes how to mount CIFS shares permanently. The shares might be hosted on a Windows computer/server, or on a Linux/UNIX server running Samba . This document also applies to SMBFS shares, which are similar to CIFS but are deprecated and should be avoided if possible ( link ). (This document does not describe how to host the shares yourself, only how to access shares that are hosted somewhere else. For hosting shares, use Samba .)","title":"Samba"},{"location":"smb2/#prerequisites","text":"We're assuming that: - Network connections have been configured properly. - Your local (Ubuntu) username is ubuntuusername . - Share username on Windows computer is msusername . - Share password on Windows computer is mspassword . - The Windows computer's name is servername this can be either an IP address or an assigned name). - The name of the share is sharename . - You want to mount the share in /media/windowsshare .","title":"Prerequisites"},{"location":"smb2/#cifs-installation","text":"On older systems:","title":"CIFS installation"},{"location":"smb2/#mounting-unprotected-guest-network-folders","text":"First, let's create the mount directory. You will need a separate directory for each mount. Then edit your /etc/fstab file (with root privileges) to add this line: Where - guest indicates you don't need a password to access the share, - uid=1000 makes the Linux user specified by the id the owner of the mounted share, allowing them to rename files, - iocharset=utf8 allows access to files with names in non-English languages. This doesn't work with shares of devices like the Buffalo Tera Station, or Windows machines that export their shares using ISO8895-15. - If there is any space in the server path , you need to replace it by \\040, for example //servername/My\\040Documents` After you add the entry to /etc/fstab type: This will (re)mount all entries listed in /etc/fstab.","title":"Mounting unprotected (guest) network folders"},{"location":"smb2/#mount-password-protected-network-folders","text":"The quickest way to auto-mounting a password-protected share is to edit /etc/fstab (with root privileges), to add this line: This is not a good idea however: /etc/fstab is readable by everyone and so is your Windows password in it. The way around this is to use a credentials file. This is a file that contains just the username and password. Using a text editor, create a file for your remote servers logon credential: Enter your Windows username and password in the file: Save the file, exit the editor. Change the permissions of the file to prevent unwanted access to your credentials: Then edit your /etc/fstab file (with root privileges) to add this line (replacing the insecure line in the example above, if you added it): Save the file, exit the editor. Finally, test the fstab entry by issuing: If there are no errors, you should test how it works after a reboot. Your remote share should mount automatically.","title":"Mount password protected network folders"},{"location":"smb2/#special-permissions","text":"If you need special permission (like chmod etc.), you'll need to add a uid (short for 'user id') or gid (for 'group id') parameter to the share's mount options.","title":"Special permissions"},{"location":"smb2/#mount-password-protected-shares-using-libpam_mount-ubuntu-904","text":"In addition to the initial assumptions, we're assuming that - Your username and password are the same on the Ubuntu machine and on the network drive. Install libpam-mount: Edit /etc/security/pam_mount.conf.xml using your preferred text editor. First, we're moving the user specific config bits to a file which users can actually edit themselves: remove the commenting tags ( ) surrounding the section called . Save the file when done. With this in place, users can create their own \\~/.pam_mount.conf.xml. Add the following:","title":"Mount password protected shares using libpam_mount (Ubuntu 9.04)"},{"location":"smb2/#troubleshooting","text":"","title":"Troubleshooting"},{"location":"smb2/#login-errors","text":"If you get the error \"mount error(13) permission denied\", then the server denied your access. Here are the first things to check: - Are you using a valid username and password? Does that account really have access to this folder? - Do you have whitespace in your credentials file? It should be , not . - Do you need a domain? For example, if you are told that your username is , then actually your username is and your domain is . The fstab entry should read: Or: - The security and version settings are interrelated. SMB1 is insecure and no longer supported by default. At first, try to not specify either security or version: do not specify or . If you still have authentication errors then you may need to specify either or or both. You can try the options listed at the [ mount.cifs man page ](http://manpages.ubuntu.com/manpages/raring/en/man8/mount.cifs.8.html \"wikilink\") . The man page list leaves out the option for some reason, but you should try that one as well (`[`see `discussion ](https://bugs.launchpad.net/ubuntu/+source/cifs-utils/+bug/1113395 \"wikilink\") ).","title":"Login errors"},{"location":"smb2/#unprotected-network-folder-wont-automount","text":"I've had a situation where an unprotected network folder wouldn't automount during bootup, but after manually entering \"sudo mount -a\" was mounted correctly. I solved this by replacing the \"guest\" option by \"username=guest,password=\". If anyone has an explanation for this, please leave a comment.","title":"Unprotected network folder won't automount"},{"location":"smb2/#mount-during-login-instead-of-boot","text":"If for some reason/etc/rc0.d/S31umountnfs.sh (networking problems for example) the automatic mounting during boot doesn't work, you can add the \"noauto\" parameter to your smbfs fstab entry and then have the share mounted at login. In /etc/fstab: In /etc/rc.local:","title":"Mount during login instead of boot"},{"location":"smb2/#slow-shutdown-due-to-a-cifsnetwork-manager-bug","text":"If you use Network Manager, and are getting really slow shutdowns, it's probably because NM shuts down before unmounting the network shares. That will cause CIFS to hang and wait for 60 seconds or so. Here's how to fix it:/etc/rc0.d/S31umountnfs.sh Ubuntu 12.04 already runs umountnfs.sh at reboot and shutdown by default (/etc/rc0.d/S31umountnfs.sh and /etc/rc6.d/S31umountnfs.sh) so this is no longer necessary.","title":"Slow shutdown due to a CIFS/Network Manager bug"},{"location":"smb2/#cifs-options-deprecated","text":"20 Feb 2008 TW Using dmask or fmask in the fstab file produces the following warnings: WARNING: CIFS mount option 'dmask' is deprecated. Use 'dir_mode' instead. WARNING: CIFS mount option 'fmask' is deprecated. Use 'file_mode' instead. Instead use this format: file_mode=0777,dir_mode=0777 . Or in some cases you might need to use file_mode=0777,dir_mode=0777,nounix ( see discussion )","title":"CIFS Options Deprecated"},{"location":"smb2/#use-of-tilde-in-pathnames-such-as-credentialssmbcredentials","text":"20 Feb 2008 TW Curiously, using credentials=\\~/.smbcredentials in fstab didn't work. I had to use the full path, i.e. /home/username/.smbcredentials (This is likely because the tilde \"\\~\" is only a shell short-hand alias for \"$HOME\"; it isn't something recognized system-wide by all programs, especially not in a system file table where the concept of \"HOME\" doesn't really exist. -Ian!) CategoryDocumentation","title":"\\== Use of tilde in pathnames such as \"credentials=\\~/.smbcredentials\""},{"location":"softwarearchitecture/","text":"software architecture Software Architecture: The Hard Parts Modern Trade-Off Analyses for Distributed Architectures Introduction There are no easy decisions in software architecture. Instead, there are many hard parts -- difficult problems or issues with no best practices -- that force you to choose among various compromises. The best design an architect can create is the least worst collection of trade-offs -- no single architecture characteristic excels as it would alone, but the balance of all the competing architecture characteristics promote project success. The Importance of Data in Architecture It has been said that data is the most important asset in a company. Businesses want to extract value from the data that they have and are finding new ways to deploy data in decision making. Operational data Data that is used to run the business. This data is defined as Online Transactional Processing (OLTP) data. Involves Create, Read, Update, Delete (CRUD) operations. Analytical data Data that is used to analyze the business. This data is defined as Online Analytical Processing (OLAP) data. Involves Read operations. Architectural Decisions Records ADR : A short noun phrase containing the architecture decision Context In this section of the ADR we will add a short one- or two-sentence description of the problem, and list the alternative solutions. Decision In this section we will state the architecture decision and provide a detailed justification of the decision. Consequences In this section of the ADR we will describe any consequences after the decision is applied, and also discuss the trade-offs that were considered.","title":"softwarearchitecture"},{"location":"softwarearchitecture/#software-architecture","text":"Software Architecture: The Hard Parts Modern Trade-Off Analyses for Distributed Architectures","title":"software architecture"},{"location":"softwarearchitecture/#introduction","text":"There are no easy decisions in software architecture. Instead, there are many hard parts -- difficult problems or issues with no best practices -- that force you to choose among various compromises. The best design an architect can create is the least worst collection of trade-offs -- no single architecture characteristic excels as it would alone, but the balance of all the competing architecture characteristics promote project success.","title":"Introduction"},{"location":"softwarearchitecture/#the-importance-of-data-in-architecture","text":"It has been said that data is the most important asset in a company. Businesses want to extract value from the data that they have and are finding new ways to deploy data in decision making. Operational data Data that is used to run the business. This data is defined as Online Transactional Processing (OLTP) data. Involves Create, Read, Update, Delete (CRUD) operations. Analytical data Data that is used to analyze the business. This data is defined as Online Analytical Processing (OLAP) data. Involves Read operations.","title":"The Importance of Data in Architecture"},{"location":"softwarearchitecture/#architectural-decisions-records","text":"ADR : A short noun phrase containing the architecture decision Context In this section of the ADR we will add a short one- or two-sentence description of the problem, and list the alternative solutions. Decision In this section we will state the architecture decision and provide a detailed justification of the decision. Consequences In this section of the ADR we will describe any consequences after the decision is applied, and also discuss the trade-offs that were considered.","title":"Architectural Decisions Records"},{"location":"sprint/","text":"sprint s3 VegFru 25 classes S4 full datasets: time, results torch/tf custom data training: duration obj det coco yolox s5 doc","title":"sprint"},{"location":"sprint/#sprint","text":"","title":"sprint"},{"location":"sprint/#s3","text":"","title":"s3"},{"location":"sprint/#vegfru","text":"25 classes","title":"VegFru"},{"location":"sprint/#s4","text":"full datasets: time, results torch/tf custom data training: duration obj det coco yolox","title":"S4"},{"location":"sprint/#s5","text":"doc","title":"s5"},{"location":"ssh/","text":"ssh sudo pacman -S openssh sudo systemctl enable --now sshd If the username on your local machine matches the one on the server you are trying to connect to, you can just type: ssh host_ip_address And hit Enter. ssh your_username@host_ip_address To open SSH port on Manjaro, you can follow these steps 1: Make sure sshd (ssh daemon) is active and listening to incoming connections on the computer you are trying to access: systemctl status sshd If it\u2019s inactive, start it using: systemctl start sshd To automatically start it on boot: systemctl enable sshd Also allow incoming connections in your firewall. ssh-keygen -t rsa -b 2048 Generating public/private rsa key pair. Enter file in which to save the key (/home/username/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/username/.ssh/id_rsa. Your public key has been saved in /home/username/.ssh/id_rsa.pub. $ ssh-copy-id id@server id@server's password:","title":"ssh"},{"location":"ssh/#ssh","text":"sudo pacman -S openssh sudo systemctl enable --now sshd If the username on your local machine matches the one on the server you are trying to connect to, you can just type: ssh host_ip_address And hit Enter. ssh your_username@host_ip_address To open SSH port on Manjaro, you can follow these steps 1: Make sure sshd (ssh daemon) is active and listening to incoming connections on the computer you are trying to access: systemctl status sshd If it\u2019s inactive, start it using: systemctl start sshd To automatically start it on boot: systemctl enable sshd Also allow incoming connections in your firewall. ssh-keygen -t rsa -b 2048 Generating public/private rsa key pair. Enter file in which to save the key (/home/username/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/username/.ssh/id_rsa. Your public key has been saved in /home/username/.ssh/id_rsa.pub. $ ssh-copy-id id@server id@server's password:","title":"ssh"},{"location":"stable_diffusion/","text":"stable diffusion negative prompt \"longbody, lowres, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality\" Examples Source Input Output (no preprocessor) (no preprocessor)","title":"stable_diffussion"},{"location":"stable_diffusion/#stable-diffusion","text":"","title":"stable diffusion"},{"location":"stable_diffusion/#negative-prompt","text":"\"longbody, lowres, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality\"","title":"negative prompt"},{"location":"stable_diffusion/#examples","text":"Source Input Output (no preprocessor) (no preprocessor)","title":"Examples"},{"location":"sway/","text":"sway You may combine output commands into one, like so: output HDMI-A-1 mode 1920x1080 pos 1920 0 bg ~/wallpaper.png stretch You can get a list of output names with swaymsg -t get_outputs. You may also match any output by using the output name \"*\". Additionally, \"-\" can be used to match the focused output by name and \"--\" can be used to match the focused output by its identifier. Some outputs may have different names when disconnecting and reconnecting. To identify these, the name can be substituted for a string consisting of the make, model and serial which you can get from swaymsg -t get_outputs. Each value must be separated by one space. For example: output \"Some Company ABC123 0x00000000\" pos 1920 0 swaymsg -t get_outputs output mode|resolution|res [--custom] [@ [Hz]] Configures the specified output to use the given mode. Modes are a combination of width and height (in pixels) and a refresh rate that your display can be configured to use. For a list of available modes for each output, use swaymsg -t get_outputs. To set a custom mode not listed in the list of available modes, use --custom. You should probably only use this if you know what you're doing. Examples: output HDMI-A-1 mode 1920x1080 output HDMI-A-1 mode 1920x1080@60Hz output position|pos Places the specified output at the specific position in the global coordinate space. The cursor may only be moved between immediately adjacent outputs. If scaling is active, it has to be considered when positioning. For example, if the scaling factor for the left output is 2, the relative position for the right output has to be divided by 2. The reference point is the top left corner so if you want the bottoms aligned this has to be considered as well. Example: output HDMI1 scale 2 output HDMI1 pos 0 1020 res 3200x1800 output eDP1 pos 1600 0 res 1920x1080 Note that the left x-pos of eDP1 is 1600 = 3200/2 and the bottom y-pos is 1020 + (1800 / 2) = 1920 = 0 + 1920 output scale Scales the specified output by the specified scale factor. An integer is recommended, but fractional values are also supported. If a fractional value are specified, be warned that it is not possible to faithfully represent the contents of your windows - they will be rendered at the next highest integer scale factor and downscaled. You may be better served by setting an integer scale factor and adjusting the font size of your applications to taste. HiDPI isn't supported with Xwayland clients (windows will blur). output scale_filter linear|nearest|smart Indicates how to scale application buffers that are rendered at a scale lower than the output's configured scale, such as lo-dpi applications on hi-dpi screens. Linear is smoother and blurrier, nearest (also known as nearest neighbor) is sharper and blockier. Setting \"smart\" will apply nearest scaling when the output has an integer scale factor, otherwise linear. The default is \"smart\". output subpixel rgb|bgr|vrgb|vbgr|none Manually sets the subpixel hinting for the specified output. This value is usually auto-detected, but some displays may misreport their subpixel geometry. Using the correct subpixel hinting allows for sharper text. Incorrect values will result in blurrier text. When changing this via swaymsg, some applications may need to be restarted to use the new value. output background|bg [ ] Sets the wallpaper for the given output to the specified file, using the given scaling mode (one of \"stretch\", \"fill\", \"fit\", \"center\", \"tile\"). If the specified file cannot be accessed or if the image does fill the entire output, a fallback color may be provided to cover the rest of the output. fallback_color should be specified as #RRGGBB. Alpha is not supported. output background|bg solid_color Sets the background of the given output to the specified color. color should be specified as #RRGGBB. Alpha is not supported. output transform [clockwise|anticlockwise] Sets the background transform to the given value. Can be one of \"90\", \"180\", \"270\" for rotation; or \"flipped\", \"flipped-90\", \"flipped-180\", \"flipped-270\" to apply a rotation and flip, or \"normal\" to apply no transform. If a single output is chosen and a rotation direction is specified (clockwise or anticlockwise) then the transform is added or subtracted from the current transform. output disable|enable Enables or disables the specified output (all outputs are enabled by default). output toggle Toggle the specified output. output dpms on|off Enables or disables the specified output via DPMS. To turn an output off (ie. blank the screen but keep workspaces as-is), one can set DPMS to off. output max_render_time off| When set to a positive number of milliseconds, enables delaying output rendering to reduce latency. The rendering is delayed in such a way as to leave the specified number of milliseconds before the next presentation for rendering. The output rendering normally takes place immediately after a presentation (vblank, buffer flip, etc.) and the frame callbacks are sent to surfaces immediately after the rendering to give surfaces the most time to draw their next frame. This results in slightly below 2 frames of latency between the surface rendering and committing new contents, and the contents being shown on screen, on average. When the output rendering is delayed, the frame callbacks are sent immediately after presentation, and the surfaces have a small timespan (1 / (refresh rate) - max_render_time) to render and commit new contents to be shown on the next presentation, resulting in below 1 frame of latency. To set this up for optimal latency: 1. Launch some full-screen application that renders continuously, like glxgears. 2. Start with max_render_time 1. Increment by 1 if you see frame drops. To achieve even lower latency, see the max_render_time surface property in sway(5). Note that this property has an effect only on backends which report the presentation timestamp and the predicted output refresh rate\u2014the DRM and the Wayland backends. Furthermore, under the Wayland backend the optimal max_render_time value may vary based on the parent compositor rendering timings. .config/sway/config.d/output.conf output DP-1 resolution 3440x1440@144Hz position 0,0 adaptive_sync off output DP-4 resolution 2560x1440@59.951Hz transform 270 position 3440,0 .config/sway/definitions.d/custom.conf set $idle_timeout 240 set $locking_timeout 1200 set $screen_timeout 1200 set $swayidle swayidle -w \\ timeout $idle_timeout 'light -G > /tmp/brightness && light -S 10' resume 'light -S $([ -f /tmp/brightness ] && cat /tmp/brightness || echo 100%)' \\ timeout $locking_timeout 'exec $locking' \\ timeout $screen_timeout 'swaymsg \"output * dpms off\"' \\ resume 'swaymsg \"output * dpms on\"' \\ before-sleep 'playerctl pause' \\ before-sleep 'exec $locking' .config/waybar/config.jsonc // ============================================================================= // // Waybar configuration // // Configuration reference: https://github.com/Alexays/Waybar/wiki/Configuration // // ============================================================================= { \"include\": [ \"/usr/share/sway/templates/waybar/config.jsonc\" ], // ------------------------------------------------------------------------- // Global configuration // ------------------------------------------------------------------------- \"layer\": \"top\", // If height property would be not present, it'd be calculated dynamically \"height\": 30, \"position\": \"top\", \"modules-left\": [\"custom/menu\", \"sway/workspaces\", \"custom/scratchpad\"], \"modules-center\": [\"custom/wf-recorder\", \"sway/mode\", \"custom/weather\"], \"modules-right\": [ // informational \"sway/language\", \"custom/github\", \"custom/clipboard\", \"custom/zeit\", \"cpu\", \"memory\", \"battery\", \"temperature\", // connecting \"network\", \"bluetooth\", // media \"custom/playerctl\", \"idle_inhibitor\", \"custom/dnd\", \"pulseaudio\", \"backlight\", // system \"custom/adaptive-light\", \"custom/sunset\", \"custom/pacman\", \"tray\", \"clock\" ], // ------------------------------------------------------------------------- // Modules // ------------------------------------------------------------------------- \"battery\": { \"interval\": 30, \"states\": { \"warning\": 30, \"critical\": 15 }, \"format-charging\": \"\uf583 {capacity}%\", \"format\": \"{icon} {capacity}%\", \"format-icons\": [\"\uf582\", \"\uf579\", \"\uf57a\", \"\uf57d\", \"\uf57f\", \"\uf578\"], \"tooltip\": true }, \"clock\": { \"interval\": 60, \"format\": \"{:%e %b %Y %H:%M}\", \"tooltip\": true, \"tooltip-format\": \"{:%B %Y}\\n{calendar}\", \"on-click\": \"swaymsg exec \\\\$calendar\" }, \"cpu\": { \"interval\": 5, \"format\": \"\ufb19 {usage}%\", \"states\": { \"warning\": 70, \"critical\": 90 }, \"on-click\": \"swaymsg exec \\\\$term_float htop\" }, \"memory\": { \"interval\": 5, \"format\": \"\uf85a {}%\", \"states\": { \"warning\": 70, \"critical\": 90 }, \"on-click\": \"swaymsg exec \\\\$term_float htop\" }, \"network\": { \"interval\": 5, \"format-wifi\": \"\uf1eb \", \"format-ethernet\": \"\uf6ff\", \"format-disconnected\": \"\ufaa9\", \"tooltip-format\": \"{ifname} ({essid}): {ipaddr}\", \"on-click\": \"swaymsg exec \\\\$term_float nmtui\" }, \"sway/mode\": { \"format\": \"{}\", \"tooltip\": false }, \"idle_inhibitor\": { \"format\": \"{icon}\", \"format-icons\": { \"activated\": \"\uf9b2\", \"deactivated\": \"\uf9b1\" }, \"tooltip\": true, \"tooltip-format-activated\": \"power-saving disabled\", \"tooltip-format-deactivated\": \"power-saving enabled\" }, \"backlight\": { \"format\": \"{icon} {percent}%\", \"format-icons\": [\"\uf5dd\", \"\uf5de\", \"\uf5df\"], \"on-scroll-up\": \"swaymsg exec \\\\$brightness_up\", \"on-scroll-down\": \"swaymsg exec \\\\$brightness_down\" }, \"pulseaudio\": { \"scroll-step\": 5, \"format\": \"{icon} {volume}%{format_source}\", \"format-muted\": \"\ufa80 {format_source}\", \"format-source\": \"\", \"format-source-muted\": \" \uf86c\", \"format-icons\": { \"headphone\": \"\uf7ca\", \"headset\": \"\uf7cd\", \"default\": [\"\ufa7e\", \"\ufa7f\", \"\ufa7d\"] }, \"tooltip-format\": \"{icon}\u200a{volume}% {format_source}\", \"on-click\": \"swaymsg exec \\\\$pulseaudio\", \"on-click-middle\": \"swaymsg exec \\\\$volume_mute\", \"on-scroll-up\": \"swaymsg exec \\\\$volume_up\", \"on-scroll-down\": \"swaymsg exec \\\\$volume_down\" }, \"temperature\": { \"critical-threshold\": 90, \"interval\": 5, \"format\": \"{icon} {temperatureC}\u00b0\", \"format-icons\": [\"\uf2cb\", \"\uf2c9\", \"\uf2c8\"], \"tooltip\": false, \"on-click\": \"swaymsg exec \\\"\\\\$term_float watch sensors\\\"\" }, \"tray\": { \"icon-size\": 21, \"spacing\": 5 }, \"custom/pacman\": { \"format\": \"\uf53b {}\", \"interval\": 3600, \"exec-if\": \"[ $(pamac checkupdates -q | wc -l) -gt 0 ]\", \"exec\": \"pamac checkupdates -q | wc -l\", \"on-click\": \"pamac-manager --updates; pkill -RTMIN+4 waybar\", \"signal\": 4 }, \"custom/menu\": { \"format\": \"\uf312\", \"on-click\": \"swaymsg exec \\\\$menu\", \"tooltip\": false }, \"bluetooth\": { \"format\": \"\uf5ae\", \"format-disabled\": \"\uf5b1\", \"on-click\": \"swaymsg exec \\\\$bluetooth\", \"on-click-right\": \"rfkill toggle bluetooth\", \"tooltip-format\": \"{}\" }, \"sway/language\": { \"format\": \"\uf11c {}\", \"min-length\": 5, \"tooltip\": false, \"on-click\": \"swaymsg input $(swaymsg -t get_inputs --raw | jq '[.[] | select(.type == \\\"keyboard\\\")][0] | .identifier') xkb_switch_layout next\" }, \"custom/scratchpad\": { \"interval\": \"once\", \"return-type\": \"json\", \"format\": \"{icon}\", \"format-icons\": { \"one\": \"\ufaae\", \"many\": \"\ufab1\" }, \"exec\": \"/bin/sh /usr/share/sway/scripts/scratchpad.sh\", \"on-click\": \"swaymsg 'scratchpad show'\", \"signal\": 7 }, \"custom/sunset\": { \"interval\": \"once\", \"tooltip\": true, \"return-type\": \"json\", \"format\": \"{icon}\", \"format-icons\": { \"on\": \"\uf834\", \"off\": \"\uf400\" }, \"exec\": \"fallback_latitude=50.1 fallback_longitude=8.7 latitude= longitude= /usr/share/sway/scripts/sunset.sh\", \"on-click\": \"/usr/share/sway/scripts/sunset.sh toggle; pkill -RTMIN+6 waybar\", \"exec-if\": \"/usr/share/sway/scripts/sunset.sh check\", \"signal\": 6 }, \"custom/wf-recorder\": { \"interval\": \"once\", \"return-type\": \"json\", \"format\": \"{}\", \"tooltip-format\": \"{tooltip}\", \"exec\": \"echo '{\\\"class\\\": \\\"recording\\\",\\\"text\\\":\\\"\uf949\\\",\\\"tooltip\\\":\\\"press $mod+Esc to stop recording\\\"}'\", \"exec-if\": \"pgrep wf-recorder\", \"on-click\": \"killall -s SIGINT wf-recorder\", \"signal\": 8 }, \"custom/github\": { \"interval\": 300, \"tooltip\": false, \"return-type\": \"json\", \"format\": \"\uf408 {}\", \"exec\": \"gh api '/notifications' -q '{ text: length }' | cat -\", \"exec-if\": \"[ -x \\\"$(command -v gh)\\\" ] && gh auth status 2>&1 | grep -q -m 1 'Logged in' && gh api '/notifications' -q 'length' | grep -q -m 1 '0' ; test $? -eq 1\", \"on-click\": \"xdg-open https://github.com/notifications && sleep 30 && pkill -RTMIN+4 waybar\", \"signal\": 4 }, \"custom/playerctl\": { \"interval\": \"once\", \"tooltip\": true, \"return-type\": \"json\", \"format\": \"{icon}\", \"format-icons\": { \"Playing\": \"\uf8e5\", \"Paused\": \"\uf90c\" }, \"exec\": \"playerctl metadata --format '{\\\"alt\\\": \\\"{{status}}\\\", \\\"tooltip\\\": \\\"{{playerName}}: {{markup_escape(title)}} - {{markup_escape(artist)}}\\\" }'\", \"on-click\": \"playerctl play-pause; pkill -RTMIN+5 waybar\", \"on-click-right\": \"playerctl next; pkill -RTMIN+5 waybar\", \"on-scroll-up\": \"playerctl position 10+; pkill -RTMIN+5 waybar\", \"on-scroll-down\": \"playerctl position 10-; pkill -RTMIN+5 waybar\", \"signal\": 5 }, \"custom/clipboard\": { \"format\": \"\uf691\", \"interval\": \"once\", \"return-type\": \"json\", \"on-click\": \"swaymsg -q exec '$clipboard'; pkill -RTMIN+9 waybar\", \"on-click-right\": \"swaymsg -q exec '$clipboard-del'; pkill -RTMIN+9 waybar\", \"on-click-middle\": \"rm -f ~/.cache/cliphist/db; pkill -RTMIN+9 waybar\", \"exec\": \"printf '{\\\"tooltip\\\":\\\"%s\\\"}' $(cliphist list | wc -l)' item(s) in the clipboard\\r(Mid click to clear)'\", \"exec-if\": \"[ -x \\\"$(command -v cliphist)\\\" ] && [ $(cliphist list | wc -l) -gt 0 ]\", \"signal\": 9 }, \"custom/weather\": { \"icon-size\": 42, \"format\": \"{icon} {}\", \"tooltip\": true, \"interval\": 3600, // accepts -c/--city -t/--temperature -d/--distance \"exec\": \"/usr/share/sway/scripts/weather.py -c sg\", \"return-type\": \"json\", \"format-icons\": { \"Unknown\": \"\ue370\", \"Cloudy\": \"\ufa8f\", \"Fog\": \"\ue313\", \"HeavyRain\": \"\ue318\", \"HeavyShowers\": \"\ue319\", \"HeavySnow\": \"\ue35e\", \"HeavySnowShowers\": \"\ufc15\", \"LightRain\": \"\ue306\", \"LightShowers\": \"\ue309\", \"LightSleet\": \"\ue3ad\", \"LightSleetShowers\": \"\ue31a\", \"LightSnow\": \"\ue31a\", \"LightSnowShowers\": \"\ufb7d\", \"PartlyCloudy\": \"\ue30c\", \"Sunny\": \"\ue30d\", \"ThunderyHeavyRain\": \"\ufb7c\", \"ThunderyShowers\": \"\ue30e\", \"ThunderySnowShowers\": \"\ue366\", \"VeryCloudy\": \"\ue33d\" } }, \"custom/zeit\": { \"return-type\": \"json\", \"interval\": \"once\", \"format\": \"{icon}\", \"format-icons\": { \"tracking\": \"\ufab4\", \"stopped\": \"\uf236\" }, \"exec\": \"/bin/sh /usr/share/sway/scripts/zeit.sh status\", \"on-click\": \"/bin/sh /usr/share/sway/scripts/zeit.sh click; pkill -RTMIN+10 waybar\", \"exec-if\": \"[ -x \\\"$(command -v zeit)\\\" ]\", \"signal\": 10 }, \"custom/dnd\": { \"interval\": \"once\", \"return-type\": \"json\", \"format\": \"{}{icon}\", \"format-icons\": { \"default\": \"\uf868\", \"dnd\": \"\ufba1\" }, \"on-click\": \"makoctl mode | grep 'do-not-disturb' && makoctl mode -r do-not-disturb || makoctl mode -a do-not-disturb; pkill -RTMIN+11 waybar\", \"on-click-right\": \"makoctl restore\", \"exec\": \"printf '{\\\"alt\\\":\\\"%s\\\",\\\"tooltip\\\":\\\"mode: %s\\\"}' $(makoctl mode | grep -q 'do-not-disturb' && echo dnd || echo default) $(makoctl mode | tail -1)\", \"signal\": 11 }, \"custom/adaptive-light\": { \"interval\": \"once\", \"tooltip\": true, \"return-type\": \"json\", \"format\": \"{icon}\", \"format-icons\": { \"on\": \"\uf5e0\", \"off\": \"\uf5df\" }, \"exec\": \"/usr/share/sway/scripts/wluma.sh\", \"on-click\": \"/usr/share/sway/scripts/wluma.sh toggle; pkill -RTMIN+12 waybar\", \"exec-if\": \"/usr/share/sway/scripts/wluma.sh check\", \"signal\": 12 } }","title":"sway"},{"location":"sway/#sway","text":"You may combine output commands into one, like so: output HDMI-A-1 mode 1920x1080 pos 1920 0 bg ~/wallpaper.png stretch You can get a list of output names with swaymsg -t get_outputs. You may also match any output by using the output name \"*\". Additionally, \"-\" can be used to match the focused output by name and \"--\" can be used to match the focused output by its identifier. Some outputs may have different names when disconnecting and reconnecting. To identify these, the name can be substituted for a string consisting of the make, model and serial which you can get from swaymsg -t get_outputs. Each value must be separated by one space. For example: output \"Some Company ABC123 0x00000000\" pos 1920 0 swaymsg -t get_outputs","title":"sway"},{"location":"sway/#output-moderesolutionres-custom-hz","text":"Configures the specified output to use the given mode. Modes are a combination of width and height (in pixels) and a refresh rate that your display can be configured to use. For a list of available modes for each output, use swaymsg -t get_outputs. To set a custom mode not listed in the list of available modes, use --custom. You should probably only use this if you know what you're doing. Examples: output HDMI-A-1 mode 1920x1080 output HDMI-A-1 mode 1920x1080@60Hz","title":"output mode|resolution|res [--custom] [@[Hz]]"},{"location":"sway/#output-positionpos","text":"Places the specified output at the specific position in the global coordinate space. The cursor may only be moved between immediately adjacent outputs. If scaling is active, it has to be considered when positioning. For example, if the scaling factor for the left output is 2, the relative position for the right output has to be divided by 2. The reference point is the top left corner so if you want the bottoms aligned this has to be considered as well. Example: output HDMI1 scale 2 output HDMI1 pos 0 1020 res 3200x1800 output eDP1 pos 1600 0 res 1920x1080 Note that the left x-pos of eDP1 is 1600 = 3200/2 and the bottom y-pos is 1020 + (1800 / 2) = 1920 = 0 + 1920","title":"output position|pos"},{"location":"sway/#output-scale","text":"Scales the specified output by the specified scale factor. An integer is recommended, but fractional values are also supported. If a fractional value are specified, be warned that it is not possible to faithfully represent the contents of your windows - they will be rendered at the next highest integer scale factor and downscaled. You may be better served by setting an integer scale factor and adjusting the font size of your applications to taste. HiDPI isn't supported with Xwayland clients (windows will blur).","title":"output scale"},{"location":"sway/#output-scale_filter-linearnearestsmart","text":"Indicates how to scale application buffers that are rendered at a scale lower than the output's configured scale, such as lo-dpi applications on hi-dpi screens. Linear is smoother and blurrier, nearest (also known as nearest neighbor) is sharper and blockier. Setting \"smart\" will apply nearest scaling when the output has an integer scale factor, otherwise linear. The default is \"smart\".","title":"output scale_filter linear|nearest|smart"},{"location":"sway/#output-subpixel-rgbbgrvrgbvbgrnone","text":"Manually sets the subpixel hinting for the specified output. This value is usually auto-detected, but some displays may misreport their subpixel geometry. Using the correct subpixel hinting allows for sharper text. Incorrect values will result in blurrier text. When changing this via swaymsg, some applications may need to be restarted to use the new value.","title":"output subpixel rgb|bgr|vrgb|vbgr|none"},{"location":"sway/#output-backgroundbg","text":"Sets the wallpaper for the given output to the specified file, using the given scaling mode (one of \"stretch\", \"fill\", \"fit\", \"center\", \"tile\"). If the specified file cannot be accessed or if the image does fill the entire output, a fallback color may be provided to cover the rest of the output. fallback_color should be specified as #RRGGBB. Alpha is not supported.","title":"output background|bg []"},{"location":"sway/#output-backgroundbg-solid_color","text":"Sets the background of the given output to the specified color. color should be specified as #RRGGBB. Alpha is not supported.","title":"output background|bg solid_color"},{"location":"sway/#output-transform-clockwiseanticlockwise","text":"Sets the background transform to the given value. Can be one of \"90\", \"180\", \"270\" for rotation; or \"flipped\", \"flipped-90\", \"flipped-180\", \"flipped-270\" to apply a rotation and flip, or \"normal\" to apply no transform. If a single output is chosen and a rotation direction is specified (clockwise or anticlockwise) then the transform is added or subtracted from the current transform.","title":"output transform [clockwise|anticlockwise]"},{"location":"sway/#output-disableenable","text":"Enables or disables the specified output (all outputs are enabled by default).","title":"output disable|enable"},{"location":"sway/#output-toggle","text":"Toggle the specified output.","title":"output toggle"},{"location":"sway/#output-dpms-onoff","text":"Enables or disables the specified output via DPMS. To turn an output off (ie. blank the screen but keep workspaces as-is), one can set DPMS to off.","title":"output dpms on|off"},{"location":"sway/#output-max_render_time-off","text":"When set to a positive number of milliseconds, enables delaying output rendering to reduce latency. The rendering is delayed in such a way as to leave the specified number of milliseconds before the next presentation for rendering. The output rendering normally takes place immediately after a presentation (vblank, buffer flip, etc.) and the frame callbacks are sent to surfaces immediately after the rendering to give surfaces the most time to draw their next frame. This results in slightly below 2 frames of latency between the surface rendering and committing new contents, and the contents being shown on screen, on average. When the output rendering is delayed, the frame callbacks are sent immediately after presentation, and the surfaces have a small timespan (1 / (refresh rate) - max_render_time) to render and commit new contents to be shown on the next presentation, resulting in below 1 frame of latency. To set this up for optimal latency: 1. Launch some full-screen application that renders continuously, like glxgears. 2. Start with max_render_time 1. Increment by 1 if you see frame drops. To achieve even lower latency, see the max_render_time surface property in sway(5). Note that this property has an effect only on backends which report the presentation timestamp and the predicted output refresh rate\u2014the DRM and the Wayland backends. Furthermore, under the Wayland backend the optimal max_render_time value may vary based on the parent compositor rendering timings. .config/sway/config.d/output.conf output DP-1 resolution 3440x1440@144Hz position 0,0 adaptive_sync off output DP-4 resolution 2560x1440@59.951Hz transform 270 position 3440,0 .config/sway/definitions.d/custom.conf set $idle_timeout 240 set $locking_timeout 1200 set $screen_timeout 1200 set $swayidle swayidle -w \\ timeout $idle_timeout 'light -G > /tmp/brightness && light -S 10' resume 'light -S $([ -f /tmp/brightness ] && cat /tmp/brightness || echo 100%)' \\ timeout $locking_timeout 'exec $locking' \\ timeout $screen_timeout 'swaymsg \"output * dpms off\"' \\ resume 'swaymsg \"output * dpms on\"' \\ before-sleep 'playerctl pause' \\ before-sleep 'exec $locking' .config/waybar/config.jsonc // ============================================================================= // // Waybar configuration // // Configuration reference: https://github.com/Alexays/Waybar/wiki/Configuration // // ============================================================================= { \"include\": [ \"/usr/share/sway/templates/waybar/config.jsonc\" ], // ------------------------------------------------------------------------- // Global configuration // ------------------------------------------------------------------------- \"layer\": \"top\", // If height property would be not present, it'd be calculated dynamically \"height\": 30, \"position\": \"top\", \"modules-left\": [\"custom/menu\", \"sway/workspaces\", \"custom/scratchpad\"], \"modules-center\": [\"custom/wf-recorder\", \"sway/mode\", \"custom/weather\"], \"modules-right\": [ // informational \"sway/language\", \"custom/github\", \"custom/clipboard\", \"custom/zeit\", \"cpu\", \"memory\", \"battery\", \"temperature\", // connecting \"network\", \"bluetooth\", // media \"custom/playerctl\", \"idle_inhibitor\", \"custom/dnd\", \"pulseaudio\", \"backlight\", // system \"custom/adaptive-light\", \"custom/sunset\", \"custom/pacman\", \"tray\", \"clock\" ], // ------------------------------------------------------------------------- // Modules // ------------------------------------------------------------------------- \"battery\": { \"interval\": 30, \"states\": { \"warning\": 30, \"critical\": 15 }, \"format-charging\": \"\uf583 {capacity}%\", \"format\": \"{icon} {capacity}%\", \"format-icons\": [\"\uf582\", \"\uf579\", \"\uf57a\", \"\uf57d\", \"\uf57f\", \"\uf578\"], \"tooltip\": true }, \"clock\": { \"interval\": 60, \"format\": \"{:%e %b %Y %H:%M}\", \"tooltip\": true, \"tooltip-format\": \"{:%B %Y}\\n{calendar}\", \"on-click\": \"swaymsg exec \\\\$calendar\" }, \"cpu\": { \"interval\": 5, \"format\": \"\ufb19 {usage}%\", \"states\": { \"warning\": 70, \"critical\": 90 }, \"on-click\": \"swaymsg exec \\\\$term_float htop\" }, \"memory\": { \"interval\": 5, \"format\": \"\uf85a {}%\", \"states\": { \"warning\": 70, \"critical\": 90 }, \"on-click\": \"swaymsg exec \\\\$term_float htop\" }, \"network\": { \"interval\": 5, \"format-wifi\": \"\uf1eb \", \"format-ethernet\": \"\uf6ff\", \"format-disconnected\": \"\ufaa9\", \"tooltip-format\": \"{ifname} ({essid}): {ipaddr}\", \"on-click\": \"swaymsg exec \\\\$term_float nmtui\" }, \"sway/mode\": { \"format\": \"{}\", \"tooltip\": false }, \"idle_inhibitor\": { \"format\": \"{icon}\", \"format-icons\": { \"activated\": \"\uf9b2\", \"deactivated\": \"\uf9b1\" }, \"tooltip\": true, \"tooltip-format-activated\": \"power-saving disabled\", \"tooltip-format-deactivated\": \"power-saving enabled\" }, \"backlight\": { \"format\": \"{icon} {percent}%\", \"format-icons\": [\"\uf5dd\", \"\uf5de\", \"\uf5df\"], \"on-scroll-up\": \"swaymsg exec \\\\$brightness_up\", \"on-scroll-down\": \"swaymsg exec \\\\$brightness_down\" }, \"pulseaudio\": { \"scroll-step\": 5, \"format\": \"{icon} {volume}%{format_source}\", \"format-muted\": \"\ufa80 {format_source}\", \"format-source\": \"\", \"format-source-muted\": \" \uf86c\", \"format-icons\": { \"headphone\": \"\uf7ca\", \"headset\": \"\uf7cd\", \"default\": [\"\ufa7e\", \"\ufa7f\", \"\ufa7d\"] }, \"tooltip-format\": \"{icon}\u200a{volume}% {format_source}\", \"on-click\": \"swaymsg exec \\\\$pulseaudio\", \"on-click-middle\": \"swaymsg exec \\\\$volume_mute\", \"on-scroll-up\": \"swaymsg exec \\\\$volume_up\", \"on-scroll-down\": \"swaymsg exec \\\\$volume_down\" }, \"temperature\": { \"critical-threshold\": 90, \"interval\": 5, \"format\": \"{icon} {temperatureC}\u00b0\", \"format-icons\": [\"\uf2cb\", \"\uf2c9\", \"\uf2c8\"], \"tooltip\": false, \"on-click\": \"swaymsg exec \\\"\\\\$term_float watch sensors\\\"\" }, \"tray\": { \"icon-size\": 21, \"spacing\": 5 }, \"custom/pacman\": { \"format\": \"\uf53b {}\", \"interval\": 3600, \"exec-if\": \"[ $(pamac checkupdates -q | wc -l) -gt 0 ]\", \"exec\": \"pamac checkupdates -q | wc -l\", \"on-click\": \"pamac-manager --updates; pkill -RTMIN+4 waybar\", \"signal\": 4 }, \"custom/menu\": { \"format\": \"\uf312\", \"on-click\": \"swaymsg exec \\\\$menu\", \"tooltip\": false }, \"bluetooth\": { \"format\": \"\uf5ae\", \"format-disabled\": \"\uf5b1\", \"on-click\": \"swaymsg exec \\\\$bluetooth\", \"on-click-right\": \"rfkill toggle bluetooth\", \"tooltip-format\": \"{}\" }, \"sway/language\": { \"format\": \"\uf11c {}\", \"min-length\": 5, \"tooltip\": false, \"on-click\": \"swaymsg input $(swaymsg -t get_inputs --raw | jq '[.[] | select(.type == \\\"keyboard\\\")][0] | .identifier') xkb_switch_layout next\" }, \"custom/scratchpad\": { \"interval\": \"once\", \"return-type\": \"json\", \"format\": \"{icon}\", \"format-icons\": { \"one\": \"\ufaae\", \"many\": \"\ufab1\" }, \"exec\": \"/bin/sh /usr/share/sway/scripts/scratchpad.sh\", \"on-click\": \"swaymsg 'scratchpad show'\", \"signal\": 7 }, \"custom/sunset\": { \"interval\": \"once\", \"tooltip\": true, \"return-type\": \"json\", \"format\": \"{icon}\", \"format-icons\": { \"on\": \"\uf834\", \"off\": \"\uf400\" }, \"exec\": \"fallback_latitude=50.1 fallback_longitude=8.7 latitude= longitude= /usr/share/sway/scripts/sunset.sh\", \"on-click\": \"/usr/share/sway/scripts/sunset.sh toggle; pkill -RTMIN+6 waybar\", \"exec-if\": \"/usr/share/sway/scripts/sunset.sh check\", \"signal\": 6 }, \"custom/wf-recorder\": { \"interval\": \"once\", \"return-type\": \"json\", \"format\": \"{}\", \"tooltip-format\": \"{tooltip}\", \"exec\": \"echo '{\\\"class\\\": \\\"recording\\\",\\\"text\\\":\\\"\uf949\\\",\\\"tooltip\\\":\\\"press $mod+Esc to stop recording\\\"}'\", \"exec-if\": \"pgrep wf-recorder\", \"on-click\": \"killall -s SIGINT wf-recorder\", \"signal\": 8 }, \"custom/github\": { \"interval\": 300, \"tooltip\": false, \"return-type\": \"json\", \"format\": \"\uf408 {}\", \"exec\": \"gh api '/notifications' -q '{ text: length }' | cat -\", \"exec-if\": \"[ -x \\\"$(command -v gh)\\\" ] && gh auth status 2>&1 | grep -q -m 1 'Logged in' && gh api '/notifications' -q 'length' | grep -q -m 1 '0' ; test $? -eq 1\", \"on-click\": \"xdg-open https://github.com/notifications && sleep 30 && pkill -RTMIN+4 waybar\", \"signal\": 4 }, \"custom/playerctl\": { \"interval\": \"once\", \"tooltip\": true, \"return-type\": \"json\", \"format\": \"{icon}\", \"format-icons\": { \"Playing\": \"\uf8e5\", \"Paused\": \"\uf90c\" }, \"exec\": \"playerctl metadata --format '{\\\"alt\\\": \\\"{{status}}\\\", \\\"tooltip\\\": \\\"{{playerName}}: {{markup_escape(title)}} - {{markup_escape(artist)}}\\\" }'\", \"on-click\": \"playerctl play-pause; pkill -RTMIN+5 waybar\", \"on-click-right\": \"playerctl next; pkill -RTMIN+5 waybar\", \"on-scroll-up\": \"playerctl position 10+; pkill -RTMIN+5 waybar\", \"on-scroll-down\": \"playerctl position 10-; pkill -RTMIN+5 waybar\", \"signal\": 5 }, \"custom/clipboard\": { \"format\": \"\uf691\", \"interval\": \"once\", \"return-type\": \"json\", \"on-click\": \"swaymsg -q exec '$clipboard'; pkill -RTMIN+9 waybar\", \"on-click-right\": \"swaymsg -q exec '$clipboard-del'; pkill -RTMIN+9 waybar\", \"on-click-middle\": \"rm -f ~/.cache/cliphist/db; pkill -RTMIN+9 waybar\", \"exec\": \"printf '{\\\"tooltip\\\":\\\"%s\\\"}' $(cliphist list | wc -l)' item(s) in the clipboard\\r(Mid click to clear)'\", \"exec-if\": \"[ -x \\\"$(command -v cliphist)\\\" ] && [ $(cliphist list | wc -l) -gt 0 ]\", \"signal\": 9 }, \"custom/weather\": { \"icon-size\": 42, \"format\": \"{icon} {}\", \"tooltip\": true, \"interval\": 3600, // accepts -c/--city -t/--temperature -d/--distance \"exec\": \"/usr/share/sway/scripts/weather.py -c sg\", \"return-type\": \"json\", \"format-icons\": { \"Unknown\": \"\ue370\", \"Cloudy\": \"\ufa8f\", \"Fog\": \"\ue313\", \"HeavyRain\": \"\ue318\", \"HeavyShowers\": \"\ue319\", \"HeavySnow\": \"\ue35e\", \"HeavySnowShowers\": \"\ufc15\", \"LightRain\": \"\ue306\", \"LightShowers\": \"\ue309\", \"LightSleet\": \"\ue3ad\", \"LightSleetShowers\": \"\ue31a\", \"LightSnow\": \"\ue31a\", \"LightSnowShowers\": \"\ufb7d\", \"PartlyCloudy\": \"\ue30c\", \"Sunny\": \"\ue30d\", \"ThunderyHeavyRain\": \"\ufb7c\", \"ThunderyShowers\": \"\ue30e\", \"ThunderySnowShowers\": \"\ue366\", \"VeryCloudy\": \"\ue33d\" } }, \"custom/zeit\": { \"return-type\": \"json\", \"interval\": \"once\", \"format\": \"{icon}\", \"format-icons\": { \"tracking\": \"\ufab4\", \"stopped\": \"\uf236\" }, \"exec\": \"/bin/sh /usr/share/sway/scripts/zeit.sh status\", \"on-click\": \"/bin/sh /usr/share/sway/scripts/zeit.sh click; pkill -RTMIN+10 waybar\", \"exec-if\": \"[ -x \\\"$(command -v zeit)\\\" ]\", \"signal\": 10 }, \"custom/dnd\": { \"interval\": \"once\", \"return-type\": \"json\", \"format\": \"{}{icon}\", \"format-icons\": { \"default\": \"\uf868\", \"dnd\": \"\ufba1\" }, \"on-click\": \"makoctl mode | grep 'do-not-disturb' && makoctl mode -r do-not-disturb || makoctl mode -a do-not-disturb; pkill -RTMIN+11 waybar\", \"on-click-right\": \"makoctl restore\", \"exec\": \"printf '{\\\"alt\\\":\\\"%s\\\",\\\"tooltip\\\":\\\"mode: %s\\\"}' $(makoctl mode | grep -q 'do-not-disturb' && echo dnd || echo default) $(makoctl mode | tail -1)\", \"signal\": 11 }, \"custom/adaptive-light\": { \"interval\": \"once\", \"tooltip\": true, \"return-type\": \"json\", \"format\": \"{icon}\", \"format-icons\": { \"on\": \"\uf5e0\", \"off\": \"\uf5df\" }, \"exec\": \"/usr/share/sway/scripts/wluma.sh\", \"on-click\": \"/usr/share/sway/scripts/wluma.sh toggle; pkill -RTMIN+12 waybar\", \"exec-if\": \"/usr/share/sway/scripts/wluma.sh check\", \"signal\": 12 } }","title":"output max_render_time off|"},{"location":"training_pipeline/","text":"trainer.train trainer-stage train, valid, test dict_keys(['train_loss', 'valid_loss', 'valid_trues', 'valid_logits', 'valid_preds', 'valid_probs', 'valid_elapsed_time', 'val_MulticlassAccuracy', 'val_MulticlassPrecision', 'val_MulticlassRecall', 'val_MulticlassAUROC', 'val_MulticlassCalibrationError', ('val_MulticlassCalibrationError', 'train_loss')]) python tools/train.py -f exps/example/custom/nano.py -d 1 -b 8 -c yolox_nano.pth","title":"Training pipeline"},{"location":"venv/","text":"venv sudo apt install python3-venv python3 -m venv foobar source foobar/bin/activate deactivate","title":"venv"},{"location":"venv/#venv","text":"sudo apt install python3-venv python3 -m venv foobar source foobar/bin/activate deactivate","title":"venv"},{"location":"wandb/","text":"Weights & Biases Machine learning experiment tracking, dataset versioning, and model evaluation import wandb from omegaconf import DictConfig import pandas as pd class WeightsAndBiases: def __init__(self, cfg: DictConfig) -> None: self.cfg: DictConfig = cfg if cfg.debug: wandb.init(mode=\"disabled\") else: wandb.init(project=cfg.project, entity=\"peekingduck\", config=cfg) def watch(self, model) -> None: wandb.watch(model) def log(self, loss) -> None: wandb.log(loss) def log_history(self, history) -> None: selected_history = { key: history[key] for key in [ \"train_loss\", \"valid_loss\", \"valid_elapsed_time\", \"val_MulticlassAccuracy\", \"val_MulticlassPrecision\", \"val_MulticlassRecall\", \"val_MulticlassAUROC\", ] } df: pd.DataFrame = pd.DataFrame(selected_history) for row_dict in df.to_dict(orient=\"records\"): wandb.log(row_dict) def log_training_loss(self, loss) -> None: wandb.log({\"train_loss\": loss}) def log_validation_loss(self, loss) -> None: wandb.log({\"val_loss\": loss}) Your personal account has 100 GB of free storage and artifacts. Usage Pricing Storage $0.08 per GB up to 10 TB $0.06 per GB up to 100 TB $0.05 per GB up to 1000 TB Over 1000 TB, contact us Artifact tracking $0.05 per GB up to 10 TB $0.03 per GB up to 100 TB $0.02 per GB up to 1000 TB Over 1000 TB, contact us One team, up to 10 users Email and chat support 100 GB storage and artifacts tracking included. For additional storage, see prices.","title":"Model Analysis"},{"location":"wandb/#weights-biases","text":"Machine learning experiment tracking, dataset versioning, and model evaluation import wandb from omegaconf import DictConfig import pandas as pd class WeightsAndBiases: def __init__(self, cfg: DictConfig) -> None: self.cfg: DictConfig = cfg if cfg.debug: wandb.init(mode=\"disabled\") else: wandb.init(project=cfg.project, entity=\"peekingduck\", config=cfg) def watch(self, model) -> None: wandb.watch(model) def log(self, loss) -> None: wandb.log(loss) def log_history(self, history) -> None: selected_history = { key: history[key] for key in [ \"train_loss\", \"valid_loss\", \"valid_elapsed_time\", \"val_MulticlassAccuracy\", \"val_MulticlassPrecision\", \"val_MulticlassRecall\", \"val_MulticlassAUROC\", ] } df: pd.DataFrame = pd.DataFrame(selected_history) for row_dict in df.to_dict(orient=\"records\"): wandb.log(row_dict) def log_training_loss(self, loss) -> None: wandb.log({\"train_loss\": loss}) def log_validation_loss(self, loss) -> None: wandb.log({\"val_loss\": loss}) Your personal account has 100 GB of free storage and artifacts. Usage Pricing Storage $0.08 per GB up to 10 TB $0.06 per GB up to 100 TB $0.05 per GB up to 1000 TB Over 1000 TB, contact us Artifact tracking $0.05 per GB up to 10 TB $0.03 per GB up to 100 TB $0.02 per GB up to 1000 TB Over 1000 TB, contact us One team, up to 10 users Email and chat support 100 GB storage and artifacts tracking included. For additional storage, see prices.","title":"Weights & Biases"},{"location":"yolox/","text":"yolox python tools/train.py -f exps/example/custom/nano.py -d 1 -b 2 -c yolox_nano.pth 2023-04-20 08:49:15.731 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 12. 98.56714 250.26144 70.88641 9.62624] 2023-04-20 08:49:15.733 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 13. 130.93268 229.41734 91.643555 72.7481 ] 2023-04-20 08:49:16 | INFO | yolox.core.trainer:261 - epoch: 5/5, iter: 60/62, gpu mem: 0Mb, mem: 5.4Gb, iter_time: 3.179s, data_time: 2.856s, total_loss: 1.6, iou_loss: 0.6, l1_loss: 0.1, conf_loss: 0.4, cls_loss: 0.4, lr: 3.085e-04, size: 416, ETA: 0:00:06 2023-04-20 08:49:18.890 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 58. 133.07756 166.66658 5.388448 4.927013] 2023-04-20 08:49:18.890 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 6. 138.9172 213.91843 94.45138 352.15067] 2023-04-20 08:49:18.890 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 4. 218.88423 140.31888 104.04545 165.01555] 2023-04-20 08:49:18.891 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 44. 141.09167 260.7022 282.48337 301.27512] 2023-04-20 08:49:18.893 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 14. 130.99757 47.066467 56.686657 20.449068] 2023-04-20 08:49:18.895 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 64. 220.6489 303.2769 114.6341 102.452896] 2023-04-20 08:49:18.895 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 61. 256.1682 55.158962 42.048447 45.89014 ] 2023-04-20 08:49:18.895 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [24. 19.571447 57.297344 19.864103 34.632 ] 2023-04-20 08:49:18.899 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 40. 178.63289 129.9869 7.916896 7.5660305] 2023-04-20 08:49:18.901 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 66. 319.8782 183.105 111.56579 109.044 ] 2023-04-20 08:49:18.901 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 54. 269.932 236.27823 85.54 143.03235] 2023-04-20 08:49:18.901 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 49. 340.53177 187.7362 147.19244 134.72554] 2023-04-20 08:49:18.903 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 69. 283.95578 123.04812 243.99066 180.18655] 2023-04-20 08:49:18.904 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 26. 183.09532 116.72701 26.500448 15.249037] 2023-04-20 08:49:18.904 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 12. 98.56714 250.26144 70.88641 9.62624] 2023-04-20 08:49:18.905 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 13. 285.06732 229.41734 91.643555 72.7481 ] 2023-04-20 08:49:22.045 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 6. 138.0828 213.91843 94.45138 352.15067] 2023-04-20 08:49:22.045 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 58. 282.92242 166.66658 5.388448 4.927013] 2023-04-20 08:49:22.046 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 44. 242.90833 260.7022 282.48337 301.27512] 2023-04-20 08:49:22.047 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 4. 218.88423 140.31888 104.04545 165.01555] 2023-04-20 08:49:22.049 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 14. 285.00244 47.066467 56.686657 20.449068] 2023-04-20 08:49:22.050 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 64. 220.6489 303.2769 114.6341 102.452896] 2023-04-20 08:49:22.050 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 24. 292.42856 57.297344 19.864103 34.632 ] 2023-04-20 08:49:22.051 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 61. 159.83177 55.158962 42.048447 45.89014 ] 2023-04-20 08:49:22.054 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 40. 237.36711 129.9869 7.916896 7.5660305] 2023-04-20 08:49:22.056 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 49. 75.46822 187.7362 147.19244 134.72554] 2023-04-20 08:49:22.056 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 54. 269.932 236.27823 85.54 143.03235] 2023-04-20 08:49:22.058 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 69. 283.95578 123.04812 243.99066 180.18655] 2023-04-20 08:49:22.059 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 26. 183.09532 116.72701 26.500448 15.249037] 2023-04-20 08:49:22.060 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 13. 285.06732 229.41734 91.643555 72.7481 ] 2023-04-20 08:49:22 | INFO | yolox.core.trainer:364 - Save weights to ./YOLOX_outputs/nano 0%| | 0/3 [00:00 - if no OSD_TYPE is set; one of disk , activate or directory will be used based on autodetection of the current OSD bootstrap state activate - the daemon expects to be passed a block device of a ceph-disk -prepared disk (via the OSD_DEVICE environment variable); no bootstrapping will be performed directory - the daemon expects to find the OSD filesystem(s) already mounted in /var/lib/ceph/osd/ disk - the daemon expects to be passed a block device via the OSD_DEVICE environment variable prepare - the daemon expects to be passed a block device and run ceph-disk prepare to bootstrap the disk (via the OSD_DEVICE environment variable) Options for OSDs (TODO: consolidate these options between the types): JOURNAL_DIR - if provided, new OSDs will be bootstrapped to use the specified directory as a common journal area. This is usually used to store the journals for more than one OSD on a common, separate disk. This currently only applies to the directory OSD type. JOURNAL - if provided, the new OSD will be bootstrapped to use the specified journal file (if you do not wish to use the default). This is currently only supported by the directory OSD type OSD_DEVICE - mandatory for activate and disk OSD types; this specifies which block device to use as the OSD OSD_JOURNAL - optional override of the OSD journal file. this only applies to the activate and disk OSD types OSD_FORCE_EXT4 - in case the osd data on ext4 is not automatically recognized (i.e. hidden by overlayfs) you can force them by settings this to yes . Without OSD_TYPE If the operator does not specify an OSD_TYPE autodetection happens: disk is used if no bootstrapped OSD is found. activate is used if a bootstrapped OSD is found and OSD_DEVICE is also provided. directory is used if a bootstrapped OSD is found and no OSD_DEVICE is provided. Without KV backend: docker run -d --net=host \\ --pid=host \\ --privileged=true \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ ceph/daemon osd With KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon osd Ceph disk Without KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ ceph/daemon osd Using bluestore: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e OSD_BLUESTORE=1 \\ ceph/daemon osd Using dmcrypt: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e OSD_DMCRYPT=1 \\ ceph/daemon osd With KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon osd Using bluestore with KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ -e OSD_BLUESTORE=1 \\ ceph/daemon osd Using dmcrypt with KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ -e OSD_DMCRYPT=1 \\ ceph/daemon osd List of available options: OSD_DEVICE is the OSD device OSD_JOURNAL is the journal for a given OSD HOSTNAME is used to place the OSD in the CRUSH map If you do not want to use --privileged=true , please fall back on the second example. Ceph disk activate This function is balance between ceph-disk and osd directory where the operator can use ceph-disk outside of the container (directly on the host) to prepare the devices. Devices will be prepared with ceph-disk prepare , then they will get activated inside the container. A priviledged container is still required as ceph-disk needs to access /dev/. So this has minimum value compare to the ceph-disk but might fit some use cases where the operators want to prepare their devices outside of a container. docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=activate \\ ceph/daemon osd Ceph OSD directory There are a number of environment variables which are used to configure the execution of the OSD: CLUSTER is the name of the ceph cluster (defaults to ceph ) If the OSD is not already created (key, configuration, OSD data), the following environment variables will control its creation: WEIGHT is the of the OSD when it is added to the CRUSH map (default is 1.0 ) JOURNAL is the location of the journal (default is the journal file inside the OSD data directory) HOSTNAME is the name of the host; it is used as a flag when adding the OSD to the CRUSH map The old option OSD_ID is now unused. Instead, the script will scan for each directory in /var/lib/ceph/osd of the form - . To create your OSDs simply run the following command: docker exec ceph osd create . Note that we now default to dropping root privileges, so it is important to set the proper ownership for your OSD directories. The Ceph OSD runs as UID:167, GID:167, so: chown -R 167:167 /var/lib/ceph/osd/ Multiple OSDs There is a problem when attempting run run multiple OSD containers on a single docker host. See issue #19. There are two workarounds, at present: Run each OSD with the --pid=host option Run multiple OSDs within the same container To run multiple OSDs within the same container, simply bind-mount each OSD datastore directory: docker run -v /osds/1:/var/lib/ceph/osd/ceph-1 -v /osds/2:/var/lib/ceph/osd/ceph-2 Ceph OSD directory single Ceph OSD directory single has a similar design to Ceph OSD directory since they both aim to run OSD processes from an already bootstrapped directory. So we assume the OSD directory has been populated already. The major different is that Ceph OSD directory single has a much simpler implementation since it only runs a single OSD process per container. It doesn't do anything with the journal as it assumes journal's symlink was provided during the initialization sequence of the OSD. This scenario goes through the OSD directory ( /var/lib/ceph/osd ) and looks for OSDs that don't have a lock held by any other OSD. If no lock is found, the OSD process starts. If all the OSDs are already running, we gently exit 0 and explain that all the OSDs are already running. Important note : if you are aiming at running multiple OSD containers on a same machine (things that you will likely do with Ceph anyway), you must enable --pid=host . However if you are running Docker 1.12 (based on https://github.com/docker/docker/pull/22481 ), you can just share the same PID namespace for the OSD containers only using: --pid=container: . BTRFS and journal If your OSD is BTRFS and you want to use PARALLEL journal mode, you will need to run this container with --privileged set to true. Otherwise, ceph-osd will have insufficient permissions and it will revert to the slower WRITEAHEAD mode. Note Re: [ https://github.com/Ulexus/docker-ceph/issues/5 ] A user has reported a consterning (and difficult to diagnose) problem wherein the OSD crashes frequently due to Docker running out of sufficient open file handles. This is understandable, as the OSDs use a great many ports during periods of high traffic. It is, therefore, recommended that you increase the number of open file handles available to Docker. On CoreOS (and probably other systemd-based systems), you can do this by creating the a file named /etc/systemd/system/docker.service.d/limits.conf with content something like: [Service] LimitNOFILE=4096 Deploy a MDS By default, the MDS does NOT create a ceph filesystem. If you wish to have this MDS create a ceph filesystem (it will only do this if the specified CEPHFS_NAME does not already exist), you must set, at a minimum, CEPHFS_CREATE=1 . It is strongly recommended that you read the rest of this section, as well. For most people, the defaults for the following optional environment variables are fine, but if you wish to customize the data and metadata pools in which your CephFS is stored, you may override the following as you wish: CEPHFS_CREATE : Whether to create the ceph filesystem (0 = no / 1 = yes), if it doesn't exist. Defaults to 0 (no) CEPHFS_NAME : The name of the new ceph filesystem and the basis on which the later variables are created. Defaults to cephfs CEPHFS_DATA_POOL : The name of the data pool for the ceph filesystem. If it does not exist, it will be created. Defaults to ${CEPHFS_NAME}_data CEPHFS_DATA_POOL_PG : The number of placement groups for the data pool. Defaults to 8 CEPHFS_METADATA_POOL : The name of the metadata pool for the ceph filesystem. If it does not exist, it will be created. Defaults to ${CEPHFS_NAME}_metadata CEPHFS_METADATA_POOL_PG : The number of placement groups for the metadata pool. Defaults to 8 Without KV backend, run: docker run -d --net=host \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /etc/ceph:/etc/ceph \\ -e CEPHFS_CREATE=1 \\ ceph/daemon mds With KV backend, run: docker run -d --net=host \\ -e CEPHFS_CREATE=1 \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon mds List of available options: MDS_NAME is the name the MDS server (DEFAULT: mds-$(hostname)). One thing to note is that metadata servers are not machine-restricted. They are not bound by their data directories and can move around the cluster. As a result, you can run more than one MDS on a single machine. If you plan to do so, you better set this variable and do something like: mds-$(hostname)-a , mds-$(hostname)-b etc... Deploy a Rados Gateway For the Rados Gateway, we deploy it with civetweb enabled by default. However it is possible to use different CGI frontends by simply giving remote address and port. Without kv backend, run: docker run -d --net=host \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /etc/ceph:/etc/ceph \\ ceph/daemon rgw With kv backend, run: docker run -d --net=host \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon rgw List of available options: RGW_CIVETWEB_PORT is the port to which civetweb is listening on (DEFAULT: 8080) RGW_NAME : default to hostname Administration via radosgw-admin from the Docker host if the RGW_NAME variable hasn't been supplied: docker exec radosgw-admin -n client.rgw.$(hostname) -k /var/lib/ceph/radosgw/$(hostname)/keyring If otherwise, $(hostname) has to be replaced by the value of RGW_NAME . To enable an external CGI interface instead of civetweb set: RGW_REMOTE_CGI=1 RGW_REMOTE_CGI_HOST=192.168.0.1 RGW_REMOTE_CGI_PORT=9000 And run the container like this docker run -d -v /etc/ceph:/etc/ceph -v /var/lib/ceph/:/var/lib/ceph -e CEPH_DAEMON=RGW -e RGW_NAME=myrgw -p 9000:9000 -e RGW_REMOTE_CGI=1 -e RGW_REMOTE_CGI_HOST=192.168.0.1 -e RGW_REMOTE_CGI_PORT=9000 ceph/daemon Deploy a REST API This is pretty straightforward. The --net=host is not mandatory, if you don't use it do not forget to expose the RESTAPI_PORT . Only available in luminous. docker run -d --net=host \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon restapi List of available options: RESTAPI_IP is the IP address to listen on (DEFAULT: 0.0.0.0) RESTAPI_PORT is the listening port of the REST API (DEFAULT: 5000) RESTAPI_BASE_URL is the base URL of the API (DEFAULT: /api/v0.1) RESTAPI_LOG_LEVEL is the log level of the API (DEFAULT: warning) RESTAPI_LOG_FILE is the location of the log file (DEFAULT: /var/log/ceph/ceph-restapi.log) Deploy a RBD mirror This is pretty straightforward. The --net=host is not mandatory, with KV we do: docker run -d --net=host \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon rbd_mirror Without KV we do: docker run -d --net=host \\ ceph/daemon rbd_mirror Fetching software. First of I want to check that I have all the latest packages in my debian system. apt update apt upgrade Next we fetch the keys and ceph packages, in this case we download the pacific packages for buster. wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add - echo deb https://download.ceph.com/debian-pacific/ buster main | sudo tee /etc/apt/sources.list.d/ceph.list apt update apt install ceph ceph-common Last we need to download the smartmontools for our nodes. This is so we can monitor our hard drives for hardware issues. echo deb http://deb.debian.org/debian buster-backports main >> /etc/apt/sources.list apt update apt install smartmontools/buster-backports A reboot when you have installed packages is always a good thing and if you need to do some extra hardware changes this is a good place to do so. shutdown -r now Configure node 1 First we will create a ceph configuration file. sudo vi /etc/ceph/ceph.conf The most important things to specify is the id and ips of your cluster monitors. A unique cluster id that you will reuse for all your nodes. And lastly a public network range that you want your monitors to be available over. The cluster network is a good addition if you have the resources to route the recovery traffic on a backbone network. [global] fsid = {cluster uuid} mon initial members = {id1}, {id2}, {id2} mon host = {ip1}, {ip2}, {ip3} public network = {network range for your public network} cluster network = {network range for your cluster network} auth cluster required = cephx auth service required = cephx auth client required = cephx Next we create keys for admin, monitors and boostrapping our drives. These keys will then be merged with the monitor key so the initial setup will have the keys used for other operations. sudo ceph-authtool --create-keyring /tmp/monkey --gen-key -n mon. --cap mon 'allow *' sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' sudo ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' sudo ceph-authtool /tmp/monkey --import-keyring /etc/ceph/ceph.client.admin.keyring sudo ceph-authtool /tmp/monkey --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring Make the monitor key available to the ceph user so we don't get an permission error when we start our services. sudo chown ceph:ceph /tmp/monkey Next up we create a monitor map so the monitors will know of each other. The monitors keeps track on other resources but for high availability the monitors needs to know who is in charge. monmaptool --create --add {node1-id} {node1-ip} --fsid {cluster uuid} /tmp/monmap monmaptool --add {node2-id} {node2-ip} --fsid {cluster uuid} /tmp/monmap monmaptool --add {node3-id} {node3-ip} --fsid {cluster uuid} /tmp/monmap Starting a new monitor is as easy as creating a new directory, creating the filesystem for and starting the service. sudo -u ceph mkdir /var/lib/ceph/mon/ceph-{node1-id} sudo -u ceph ceph-mon --mkfs -i {node1-id} --monmap /tmp/monmap --keyring /tmp/monkey sudo systemctl start ceph-mon@{node1-id} Next up we need a manager so we could configure and monitor our cluster through a visual dashboard. First we create a new key, put that key in a newly created directory and start the service. Enabling a dashboard is as easy as running the command for enabling, creating / assigning a certificate and creating a new admin user. sudo ceph auth get-or-create mgr.{node1-id} mon 'allow profile mgr' osd 'allow *' mds 'allow *' sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-{node1-id} sudo -u ceph vi /var/lib/ceph/mgr/ceph-{node1-id}/keyring sudo systemctl start ceph-mgr@{node1-id} sudo ceph mgr module enable dashboard sudo ceph dashboard create-self-signed-cert sudo ceph dashboard ac-user-create admin -i passwd administrator Setting up more nodes. First of we need to copy over the configuration, monitor map and all the keys over to our new host. sudo scp {user}@{server}:/etc/ceph/ceph.conf /etc/ceph/ceph.conf sudo scp {user}@{server}:/etc/ceph/ceph.client.admin.keyring /etc/ceph/ceph.client.admin.keyring sudo scp {user}@{server}:/var/lib/ceph/bootstrap-osd/ceph.keyring /var/lib/ceph/bootstrap-osd/ceph.keyring sudo scp {user}@{server}:/tmp/monmap /tmp/monmap sudo scp {user}@{server}:/tmp/monkey /tmp/monkey Next up we setup the monitor node exactly as we did with the first node. sudo -u ceph mkdir /var/lib/ceph/mon/ceph-{node2-id} sudo -u ceph ceph-mon --mkfs -i {node2-id} --monmap /tmp/monmap --keyring /tmp/monkey sudo systemctl start ceph-mon@{node2-id} sudo ceph -s sudo ceph mon enable-msgr2 Then we setup the manager node exactly as we did with the first node. sudo ceph auth get-or-create mgr.{node2-id} mon 'allow profile mgr' osd 'allow *' mds 'allow *' sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-{node2-id} sudo -u ceph vi /var/lib/ceph/mgr/ceph-{node2-id}/keyring sudo systemctl start ceph-mgr@{node2-id} Adding storage When the cluster is up and running and all monitors are in qourum you could add storage services. This is easily done via the volume command. First prepare a disk so it will be known by the cluster and have the keys and configuration copied to the management directory. Next up you activate the service so your storage nodes will be ready to use. This will be done for all the harddrives you want to add to your network. sudo ceph-volume lvm prepare --data /dev/sdb sudo ceph-volume lvm activate {osd-number} {osd-uuid} Post configuration Last but not least you want to ensure that all the services starts after a reboot. In debian you do that by enabling the services. sudo systemctl enable ceph-mon@{node-id} sudo systemctl enable ceph-mgr@{node-id} sudo systemctl enable ceph-osd@{osd-number}","title":"ceph"},{"location":"ceph/#manual-install-of-a-ceph-cluster","text":"","title":"Manual install of a Ceph Cluster."},{"location":"ceph/#daemon-container","text":"This Dockerfile may be used to bootstrap a Ceph cluster with all the Ceph daemons running. To run a certain type of daemon, simply use the name of the daemon as $1 . Valid values are: mon deploys a Ceph monitor osd deploys an OSD using the method specified by OSD_TYPE osd_directory deploys one or multiple OSDs in a single container using a prepared directory (used in scenario where the operator doesn't want to use --privileged=true ) osd_directory_single deploys an single OSD per container using a prepared directory (used in scenario where the operator doesn't want to use --privileged=true ) osd_ceph_disk deploys an OSD using ceph-disk, so you have to provide a whole device (ie: /dev/sdb) mds deploys a MDS rgw deploys a Rados Gateway","title":"Daemon container"},{"location":"ceph/#usage","text":"You can use this container to bootstrap any Ceph daemon. CLUSTER is the name of the cluster (DEFAULT: ceph)","title":"Usage"},{"location":"ceph/#selinux","text":"If SELinux is enabled, run the following commands: sudo chcon -Rt svirt_sandbox_file_t /etc/ceph sudo chcon -Rt svirt_sandbox_file_t /var/lib/ceph","title":"SELinux"},{"location":"ceph/#kv-backends","text":"We currently support one KV backend to store our configuration flags, keys and maps: etcd. There is a ceph.defaults config file in the image that is used for defaults to bootstrap daemons. It will add the keys if they are not already present. You can either pre-populate the KV store with your own settings, or provide a ceph.defaults config file. To supply your own defaults, make sure to mount the /etc/ceph/ volume and place your ceph.defaults file there. Important variables in ceph.defaults to add/change when you bootstrap an OSD: /osd/osd_journal_size /osd/cluster_network /osd/public_network Note: cluster_network and public_network are currently not populated in the defaults, but can be passed as environment variables with -e CEPH_PUBLIC_NETWORK=... for more flexibility","title":"KV backends"},{"location":"ceph/#populate-key-value-store","text":"docker run -d --net=host \\ -e KV_TYPE=etcd \\ -e KV_IP=127.0.0.1 \\ -e KV_PORT=2379 \\ ceph/daemon populate_kvstore","title":"Populate Key Value store"},{"location":"ceph/#zap-a-device","text":"Sometimes you might want to destroy partition tables from a disk. For this you can use the zap_device scenario that works as follow: docker run -d --privileged=true \\ -v /dev/:/dev/ \\ -e OSD_DEVICE=/dev/sdd \\ ceph/daemon zap_device","title":"Zap a device"},{"location":"ceph/#deploy-a-monitor","text":"A monitor requires some persistent storage for the docker container. If a KV store is used, /etc/ceph will be auto-generated from data kept in the KV store. /var/lib/ceph , however, must be provided by a docker volume. The ceph mon will periodically store data into /var/lib/ceph , including the latest copy of the CRUSH map. If a mon restarts, it will attempt to download the latest monmap and CRUSH map from other peer monitors. However, if all mon daemons have gone down, monitors must be able to recover their previous maps. The docker volume used for /var/lib/ceph should be backed by some durable storage, and must be able to survive container and node restarts. Without KV store, run: docker run -d --net=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -e MON_IP=192.168.0.20 \\ -e CEPH_PUBLIC_NETWORK=192.168.0.0/24 \\ ceph/daemon mon With KV store, run: docker run -d --net=host \\ -v /var/lib/ceph:/var/lib/ceph \\ -e MON_IP=192.168.0.20 \\ -e CEPH_PUBLIC_NETWORK=192.168.0.0/24 \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon mon List of available options: MON_NAME : name of the monitor (default to hostname) CEPH_PUBLIC_NETWORK : CIDR of the host running Docker, it should be in the same network as the MON_IP CEPH_CLUSTER_NETWORK : CIDR of a secondary interface of the host running Docker. Used for the OSD replication traffic MON_IP : IP address of the host running Docker NETWORK_AUTO_DETECT : Whether and how to attempt IP and network autodetection. Meant to be used without --net=host . NEW_USER_KEYRING : if specified, it will be imported to keyrings. Works in demo mode only. 0 = Do not detect (default) 1 = Detect IPv6, fallback to IPv4 (if no globally-routable IPv6 address detected) 4 = Detect IPv4 only 6 = Detect IPv6 only","title":"Deploy a monitor"},{"location":"ceph/#deploy-a-manager-daemon","text":"Since luminous, a manager daemon is mandatory, see docs Without KV store, run: docker run -d --net=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ ceph/daemon mgr With KV store, run: docker run -d --net=host \\ -v /var/lib/ceph:/var/lib/ceph \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon mgr","title":"Deploy a Manager daemon"},{"location":"ceph/#deploy-an-osd","text":"There are four available OSD_TYPE values: - if no OSD_TYPE is set; one of disk , activate or directory will be used based on autodetection of the current OSD bootstrap state activate - the daemon expects to be passed a block device of a ceph-disk -prepared disk (via the OSD_DEVICE environment variable); no bootstrapping will be performed directory - the daemon expects to find the OSD filesystem(s) already mounted in /var/lib/ceph/osd/ disk - the daemon expects to be passed a block device via the OSD_DEVICE environment variable prepare - the daemon expects to be passed a block device and run ceph-disk prepare to bootstrap the disk (via the OSD_DEVICE environment variable) Options for OSDs (TODO: consolidate these options between the types): JOURNAL_DIR - if provided, new OSDs will be bootstrapped to use the specified directory as a common journal area. This is usually used to store the journals for more than one OSD on a common, separate disk. This currently only applies to the directory OSD type. JOURNAL - if provided, the new OSD will be bootstrapped to use the specified journal file (if you do not wish to use the default). This is currently only supported by the directory OSD type OSD_DEVICE - mandatory for activate and disk OSD types; this specifies which block device to use as the OSD OSD_JOURNAL - optional override of the OSD journal file. this only applies to the activate and disk OSD types OSD_FORCE_EXT4 - in case the osd data on ext4 is not automatically recognized (i.e. hidden by overlayfs) you can force them by settings this to yes .","title":"Deploy an OSD"},{"location":"ceph/#without-osd_type","text":"If the operator does not specify an OSD_TYPE autodetection happens: disk is used if no bootstrapped OSD is found. activate is used if a bootstrapped OSD is found and OSD_DEVICE is also provided. directory is used if a bootstrapped OSD is found and no OSD_DEVICE is provided. Without KV backend: docker run -d --net=host \\ --pid=host \\ --privileged=true \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ ceph/daemon osd With KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon osd","title":"Without OSD_TYPE"},{"location":"ceph/#ceph-disk","text":"Without KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ ceph/daemon osd Using bluestore: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e OSD_BLUESTORE=1 \\ ceph/daemon osd Using dmcrypt: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e OSD_DMCRYPT=1 \\ ceph/daemon osd With KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon osd Using bluestore with KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ -e OSD_BLUESTORE=1 \\ ceph/daemon osd Using dmcrypt with KV backend: docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=disk \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ -e OSD_DMCRYPT=1 \\ ceph/daemon osd List of available options: OSD_DEVICE is the OSD device OSD_JOURNAL is the journal for a given OSD HOSTNAME is used to place the OSD in the CRUSH map If you do not want to use --privileged=true , please fall back on the second example.","title":"Ceph disk"},{"location":"ceph/#ceph-disk-activate","text":"This function is balance between ceph-disk and osd directory where the operator can use ceph-disk outside of the container (directly on the host) to prepare the devices. Devices will be prepared with ceph-disk prepare , then they will get activated inside the container. A priviledged container is still required as ceph-disk needs to access /dev/. So this has minimum value compare to the ceph-disk but might fit some use cases where the operators want to prepare their devices outside of a container. docker run -d --net=host \\ --privileged=true \\ --pid=host \\ -v /etc/ceph:/etc/ceph \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /dev/:/dev/ \\ -v /run/udev/:/run/udev/ \\ -e OSD_DEVICE=/dev/vdd \\ -e OSD_TYPE=activate \\ ceph/daemon osd","title":"Ceph disk activate"},{"location":"ceph/#ceph-osd-directory","text":"There are a number of environment variables which are used to configure the execution of the OSD: CLUSTER is the name of the ceph cluster (defaults to ceph ) If the OSD is not already created (key, configuration, OSD data), the following environment variables will control its creation: WEIGHT is the of the OSD when it is added to the CRUSH map (default is 1.0 ) JOURNAL is the location of the journal (default is the journal file inside the OSD data directory) HOSTNAME is the name of the host; it is used as a flag when adding the OSD to the CRUSH map The old option OSD_ID is now unused. Instead, the script will scan for each directory in /var/lib/ceph/osd of the form - . To create your OSDs simply run the following command: docker exec ceph osd create . Note that we now default to dropping root privileges, so it is important to set the proper ownership for your OSD directories. The Ceph OSD runs as UID:167, GID:167, so: chown -R 167:167 /var/lib/ceph/osd/","title":"Ceph OSD directory"},{"location":"ceph/#multiple-osds","text":"There is a problem when attempting run run multiple OSD containers on a single docker host. See issue #19. There are two workarounds, at present: Run each OSD with the --pid=host option Run multiple OSDs within the same container To run multiple OSDs within the same container, simply bind-mount each OSD datastore directory: docker run -v /osds/1:/var/lib/ceph/osd/ceph-1 -v /osds/2:/var/lib/ceph/osd/ceph-2","title":"Multiple OSDs"},{"location":"ceph/#ceph-osd-directory-single","text":"Ceph OSD directory single has a similar design to Ceph OSD directory since they both aim to run OSD processes from an already bootstrapped directory. So we assume the OSD directory has been populated already. The major different is that Ceph OSD directory single has a much simpler implementation since it only runs a single OSD process per container. It doesn't do anything with the journal as it assumes journal's symlink was provided during the initialization sequence of the OSD. This scenario goes through the OSD directory ( /var/lib/ceph/osd ) and looks for OSDs that don't have a lock held by any other OSD. If no lock is found, the OSD process starts. If all the OSDs are already running, we gently exit 0 and explain that all the OSDs are already running. Important note : if you are aiming at running multiple OSD containers on a same machine (things that you will likely do with Ceph anyway), you must enable --pid=host . However if you are running Docker 1.12 (based on https://github.com/docker/docker/pull/22481 ), you can just share the same PID namespace for the OSD containers only using: --pid=container: .","title":"Ceph OSD directory single"},{"location":"ceph/#btrfs-and-journal","text":"If your OSD is BTRFS and you want to use PARALLEL journal mode, you will need to run this container with --privileged set to true. Otherwise, ceph-osd will have insufficient permissions and it will revert to the slower WRITEAHEAD mode.","title":"BTRFS and journal"},{"location":"ceph/#note","text":"Re: [ https://github.com/Ulexus/docker-ceph/issues/5 ] A user has reported a consterning (and difficult to diagnose) problem wherein the OSD crashes frequently due to Docker running out of sufficient open file handles. This is understandable, as the OSDs use a great many ports during periods of high traffic. It is, therefore, recommended that you increase the number of open file handles available to Docker. On CoreOS (and probably other systemd-based systems), you can do this by creating the a file named /etc/systemd/system/docker.service.d/limits.conf with content something like: [Service] LimitNOFILE=4096","title":"Note"},{"location":"ceph/#deploy-a-mds","text":"By default, the MDS does NOT create a ceph filesystem. If you wish to have this MDS create a ceph filesystem (it will only do this if the specified CEPHFS_NAME does not already exist), you must set, at a minimum, CEPHFS_CREATE=1 . It is strongly recommended that you read the rest of this section, as well. For most people, the defaults for the following optional environment variables are fine, but if you wish to customize the data and metadata pools in which your CephFS is stored, you may override the following as you wish: CEPHFS_CREATE : Whether to create the ceph filesystem (0 = no / 1 = yes), if it doesn't exist. Defaults to 0 (no) CEPHFS_NAME : The name of the new ceph filesystem and the basis on which the later variables are created. Defaults to cephfs CEPHFS_DATA_POOL : The name of the data pool for the ceph filesystem. If it does not exist, it will be created. Defaults to ${CEPHFS_NAME}_data CEPHFS_DATA_POOL_PG : The number of placement groups for the data pool. Defaults to 8 CEPHFS_METADATA_POOL : The name of the metadata pool for the ceph filesystem. If it does not exist, it will be created. Defaults to ${CEPHFS_NAME}_metadata CEPHFS_METADATA_POOL_PG : The number of placement groups for the metadata pool. Defaults to 8 Without KV backend, run: docker run -d --net=host \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /etc/ceph:/etc/ceph \\ -e CEPHFS_CREATE=1 \\ ceph/daemon mds With KV backend, run: docker run -d --net=host \\ -e CEPHFS_CREATE=1 \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon mds List of available options: MDS_NAME is the name the MDS server (DEFAULT: mds-$(hostname)). One thing to note is that metadata servers are not machine-restricted. They are not bound by their data directories and can move around the cluster. As a result, you can run more than one MDS on a single machine. If you plan to do so, you better set this variable and do something like: mds-$(hostname)-a , mds-$(hostname)-b etc...","title":"Deploy a MDS"},{"location":"ceph/#deploy-a-rados-gateway","text":"For the Rados Gateway, we deploy it with civetweb enabled by default. However it is possible to use different CGI frontends by simply giving remote address and port. Without kv backend, run: docker run -d --net=host \\ -v /var/lib/ceph/:/var/lib/ceph/ \\ -v /etc/ceph:/etc/ceph \\ ceph/daemon rgw With kv backend, run: docker run -d --net=host \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon rgw List of available options: RGW_CIVETWEB_PORT is the port to which civetweb is listening on (DEFAULT: 8080) RGW_NAME : default to hostname Administration via radosgw-admin from the Docker host if the RGW_NAME variable hasn't been supplied: docker exec radosgw-admin -n client.rgw.$(hostname) -k /var/lib/ceph/radosgw/$(hostname)/keyring If otherwise, $(hostname) has to be replaced by the value of RGW_NAME . To enable an external CGI interface instead of civetweb set: RGW_REMOTE_CGI=1 RGW_REMOTE_CGI_HOST=192.168.0.1 RGW_REMOTE_CGI_PORT=9000 And run the container like this docker run -d -v /etc/ceph:/etc/ceph -v /var/lib/ceph/:/var/lib/ceph -e CEPH_DAEMON=RGW -e RGW_NAME=myrgw -p 9000:9000 -e RGW_REMOTE_CGI=1 -e RGW_REMOTE_CGI_HOST=192.168.0.1 -e RGW_REMOTE_CGI_PORT=9000 ceph/daemon","title":"Deploy a Rados Gateway"},{"location":"ceph/#deploy-a-rest-api","text":"This is pretty straightforward. The --net=host is not mandatory, if you don't use it do not forget to expose the RESTAPI_PORT . Only available in luminous. docker run -d --net=host \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon restapi List of available options: RESTAPI_IP is the IP address to listen on (DEFAULT: 0.0.0.0) RESTAPI_PORT is the listening port of the REST API (DEFAULT: 5000) RESTAPI_BASE_URL is the base URL of the API (DEFAULT: /api/v0.1) RESTAPI_LOG_LEVEL is the log level of the API (DEFAULT: warning) RESTAPI_LOG_FILE is the location of the log file (DEFAULT: /var/log/ceph/ceph-restapi.log)","title":"Deploy a REST API"},{"location":"ceph/#deploy-a-rbd-mirror","text":"This is pretty straightforward. The --net=host is not mandatory, with KV we do: docker run -d --net=host \\ -e KV_TYPE=etcd \\ -e KV_IP=192.168.0.20 \\ ceph/daemon rbd_mirror Without KV we do: docker run -d --net=host \\ ceph/daemon rbd_mirror","title":"Deploy a RBD mirror"},{"location":"ceph/#fetching-software","text":"First of I want to check that I have all the latest packages in my debian system. apt update apt upgrade Next we fetch the keys and ceph packages, in this case we download the pacific packages for buster. wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add - echo deb https://download.ceph.com/debian-pacific/ buster main | sudo tee /etc/apt/sources.list.d/ceph.list apt update apt install ceph ceph-common Last we need to download the smartmontools for our nodes. This is so we can monitor our hard drives for hardware issues. echo deb http://deb.debian.org/debian buster-backports main >> /etc/apt/sources.list apt update apt install smartmontools/buster-backports A reboot when you have installed packages is always a good thing and if you need to do some extra hardware changes this is a good place to do so. shutdown -r now","title":"Fetching software."},{"location":"ceph/#configure-node-1","text":"First we will create a ceph configuration file. sudo vi /etc/ceph/ceph.conf The most important things to specify is the id and ips of your cluster monitors. A unique cluster id that you will reuse for all your nodes. And lastly a public network range that you want your monitors to be available over. The cluster network is a good addition if you have the resources to route the recovery traffic on a backbone network. [global] fsid = {cluster uuid} mon initial members = {id1}, {id2}, {id2} mon host = {ip1}, {ip2}, {ip3} public network = {network range for your public network} cluster network = {network range for your cluster network} auth cluster required = cephx auth service required = cephx auth client required = cephx Next we create keys for admin, monitors and boostrapping our drives. These keys will then be merged with the monitor key so the initial setup will have the keys used for other operations. sudo ceph-authtool --create-keyring /tmp/monkey --gen-key -n mon. --cap mon 'allow *' sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' sudo ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' sudo ceph-authtool /tmp/monkey --import-keyring /etc/ceph/ceph.client.admin.keyring sudo ceph-authtool /tmp/monkey --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring Make the monitor key available to the ceph user so we don't get an permission error when we start our services. sudo chown ceph:ceph /tmp/monkey Next up we create a monitor map so the monitors will know of each other. The monitors keeps track on other resources but for high availability the monitors needs to know who is in charge. monmaptool --create --add {node1-id} {node1-ip} --fsid {cluster uuid} /tmp/monmap monmaptool --add {node2-id} {node2-ip} --fsid {cluster uuid} /tmp/monmap monmaptool --add {node3-id} {node3-ip} --fsid {cluster uuid} /tmp/monmap Starting a new monitor is as easy as creating a new directory, creating the filesystem for and starting the service. sudo -u ceph mkdir /var/lib/ceph/mon/ceph-{node1-id} sudo -u ceph ceph-mon --mkfs -i {node1-id} --monmap /tmp/monmap --keyring /tmp/monkey sudo systemctl start ceph-mon@{node1-id} Next up we need a manager so we could configure and monitor our cluster through a visual dashboard. First we create a new key, put that key in a newly created directory and start the service. Enabling a dashboard is as easy as running the command for enabling, creating / assigning a certificate and creating a new admin user. sudo ceph auth get-or-create mgr.{node1-id} mon 'allow profile mgr' osd 'allow *' mds 'allow *' sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-{node1-id} sudo -u ceph vi /var/lib/ceph/mgr/ceph-{node1-id}/keyring sudo systemctl start ceph-mgr@{node1-id} sudo ceph mgr module enable dashboard sudo ceph dashboard create-self-signed-cert sudo ceph dashboard ac-user-create admin -i passwd administrator","title":"Configure node 1"},{"location":"ceph/#setting-up-more-nodes","text":"First of we need to copy over the configuration, monitor map and all the keys over to our new host. sudo scp {user}@{server}:/etc/ceph/ceph.conf /etc/ceph/ceph.conf sudo scp {user}@{server}:/etc/ceph/ceph.client.admin.keyring /etc/ceph/ceph.client.admin.keyring sudo scp {user}@{server}:/var/lib/ceph/bootstrap-osd/ceph.keyring /var/lib/ceph/bootstrap-osd/ceph.keyring sudo scp {user}@{server}:/tmp/monmap /tmp/monmap sudo scp {user}@{server}:/tmp/monkey /tmp/monkey Next up we setup the monitor node exactly as we did with the first node. sudo -u ceph mkdir /var/lib/ceph/mon/ceph-{node2-id} sudo -u ceph ceph-mon --mkfs -i {node2-id} --monmap /tmp/monmap --keyring /tmp/monkey sudo systemctl start ceph-mon@{node2-id} sudo ceph -s sudo ceph mon enable-msgr2 Then we setup the manager node exactly as we did with the first node. sudo ceph auth get-or-create mgr.{node2-id} mon 'allow profile mgr' osd 'allow *' mds 'allow *' sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-{node2-id} sudo -u ceph vi /var/lib/ceph/mgr/ceph-{node2-id}/keyring sudo systemctl start ceph-mgr@{node2-id}","title":"Setting up more nodes."},{"location":"ceph/#adding-storage","text":"When the cluster is up and running and all monitors are in qourum you could add storage services. This is easily done via the volume command. First prepare a disk so it will be known by the cluster and have the keys and configuration copied to the management directory. Next up you activate the service so your storage nodes will be ready to use. This will be done for all the harddrives you want to add to your network. sudo ceph-volume lvm prepare --data /dev/sdb sudo ceph-volume lvm activate {osd-number} {osd-uuid}","title":"Adding storage"},{"location":"ceph/#post-configuration","text":"Last but not least you want to ensure that all the services starts after a reboot. In debian you do that by enabling the services. sudo systemctl enable ceph-mon@{node-id} sudo systemctl enable ceph-mgr@{node-id} sudo systemctl enable ceph-osd@{osd-number}","title":"Post configuration"},{"location":"chart/","text":"chart","title":"Chart"},{"location":"chart/#chart","text":"","title":"chart"},{"location":"code/","text":"code install vscode git clone https://AUR.archlinux.org/visual-studio-code-bin.git cd visual-studio-code-bin makepkg -s sudo pacman -U *code-bin-*.pkg.tar.zst cd ../ && sudo rm -rfv visual-studio-code-bin/","title":"vscode"},{"location":"code/#code","text":"","title":"code"},{"location":"code/#install-vscode","text":"git clone https://AUR.archlinux.org/visual-studio-code-bin.git cd visual-studio-code-bin makepkg -s sudo pacman -U *code-bin-*.pkg.tar.zst cd ../ && sudo rm -rfv visual-studio-code-bin/","title":"install vscode"},{"location":"conda/","text":"conda cheat sheet conda update -n base conda conda remove --name myenv --all conda env remove -n tf2 conda init powershell conda env create -f environment.yml conda env export > environment.yml conda create --name myenv python=3.10 conda install --file requirements.txt conda install --file requirements.txt -c conda-forge conda create --name tft --clone 311 pip install -r /path/to/requirements.txt pip install --upgrade --force-reinstall -r requirements.txt python -m ipykernel install --user --name myenv --display-name \"Python (myenv)\" conda clean -a conda install -y -c apple tensorflow-deps pip install tensorflow-macos tensorflow-metal bash Miniconda3-py310_22.11.1-1-Linux-x86_64.sh conda list conda config --add channels conda-forge conda config --describe conda config --set auto_activate_base false conda update --all pip --disable-pip-version-check list --outdated --format=json | python -c \"import json, sys; print('\\n'.join([x['name'] for x in json.load(sys.stdin)]))\" | xargs -n1 pip install -U conda update conda conda install --file requirements.txt -c conda-forge pip install -r requirements.txt # archlinux echo \"[ -f /opt/miniconda3/etc/profile.d/conda.sh ] && source /opt/miniconda3/etc/profile.d/conda.sh\" >> ~/.zshrc # ubuntu mkdir -p ~/miniconda3 wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 rm -rf ~/miniconda3/miniconda.sh source ~/miniconda3/etc/profile.d/conda.sh conda init zsh","title":"Conda"},{"location":"conda/#conda-cheat-sheet","text":"conda update -n base conda conda remove --name myenv --all conda env remove -n tf2 conda init powershell conda env create -f environment.yml conda env export > environment.yml conda create --name myenv python=3.10 conda install --file requirements.txt conda install --file requirements.txt -c conda-forge conda create --name tft --clone 311 pip install -r /path/to/requirements.txt pip install --upgrade --force-reinstall -r requirements.txt python -m ipykernel install --user --name myenv --display-name \"Python (myenv)\" conda clean -a conda install -y -c apple tensorflow-deps pip install tensorflow-macos tensorflow-metal bash Miniconda3-py310_22.11.1-1-Linux-x86_64.sh conda list conda config --add channels conda-forge conda config --describe conda config --set auto_activate_base false conda update --all pip --disable-pip-version-check list --outdated --format=json | python -c \"import json, sys; print('\\n'.join([x['name'] for x in json.load(sys.stdin)]))\" | xargs -n1 pip install -U conda update conda conda install --file requirements.txt -c conda-forge pip install -r requirements.txt # archlinux echo \"[ -f /opt/miniconda3/etc/profile.d/conda.sh ] && source /opt/miniconda3/etc/profile.d/conda.sh\" >> ~/.zshrc # ubuntu mkdir -p ~/miniconda3 wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 rm -rf ~/miniconda3/miniconda.sh source ~/miniconda3/etc/profile.d/conda.sh conda init zsh","title":"conda cheat sheet"},{"location":"dataframe/","text":"Dataframe df[\"cancer\"].value_counts().plot(kind='barh') df[[\"patient_id\", \"image_id\"]].astype(str).apply(lambda x: 'data/rsna/'+'/'.join(x)+'.png', axis=1) list(mapping.keys())[list(mapping.values()).index(x)] `` import logging from configs import LOGGER_NAME logger = logging.getLogger(LOGGER_NAME) # pylint: disable=invalid-name logger.info Rows of a pandas dataframe df def __len__(self) -> int: \"\"\"Return the length of the dataset.\"\"\" return len(self.df.index) In [7]: timeit len(df.index) 1000000 loops, best of 3: 248 ns per loop In [8]: timeit len(df) 1000000 loops, best of 3: 573 ns per loop","title":"Dataframe"},{"location":"dataframe/#dataframe","text":"df[\"cancer\"].value_counts().plot(kind='barh') df[[\"patient_id\", \"image_id\"]].astype(str).apply(lambda x: 'data/rsna/'+'/'.join(x)+'.png', axis=1) list(mapping.keys())[list(mapping.values()).index(x)] `` import logging from configs import LOGGER_NAME logger = logging.getLogger(LOGGER_NAME) # pylint: disable=invalid-name logger.info","title":"Dataframe"},{"location":"dataframe/#rows-of-a-pandas-dataframe-df","text":"def __len__(self) -> int: \"\"\"Return the length of the dataset.\"\"\" return len(self.df.index) In [7]: timeit len(df.index) 1000000 loops, best of 3: 248 ns per loop In [8]: timeit len(df) 1000000 loops, best of 3: 573 ns per loop","title":"Rows of a pandas dataframe df"},{"location":"db/","text":"SQLAlchemy conda create --name sql python=3.11 pip install SQLAlchemy pip install mariadb pip install PyMySQL brew install mariadb-connector-c brew install mysql docker-compose up --build -d","title":"database"},{"location":"db/#sqlalchemy","text":"conda create --name sql python=3.11 pip install SQLAlchemy pip install mariadb pip install PyMySQL brew install mariadb-connector-c brew install mysql docker-compose up --build -d","title":"SQLAlchemy"},{"location":"docker/","text":"Docker sudo apt update && sudo apt upgrade -y sudo apt install git curl htop mc rsync zsh-autosuggestions sudo apt install zsh chsh -s $(which zsh) sh -c \"$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)\" rsync -avuPz ./.oh-my-zsh/custom/themes/custom.zsh-theme furyhawk@arm:/home/furyhawk/.oh-my-zsh/custom/themes/custom.zsh-theme git clone https://github.com/zsh-users/zsh-autosuggestions.git $ZSH_CUSTOM/plugins/zsh-autosuggestions git clone https://github.com/zdharma-continuum/fast-syntax-highlighting.git ${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/plugins/fast-syntax-highlighting ## Enable plugins by adding them to .zshrc. - Open .zshrc `nano ~/.zshrc` - ZSH_THEME=\"custom\" - Find the line which says `plugins=(git)`. - Replace that line with `plugins=(git zsh-autosuggestions zsh-syntax-highlighting fast-syntax-highlighting zsh-autocomplete)` touch .zprofile nano .zprofile # set PATH so it includes user's private bin if it exists if [ -d \"$HOME/bin\" ] ; then PATH=\"$HOME/bin:$PATH\" fi # set PATH so it includes user's private bin if it exists if [ -d \"$HOME/.local/bin\" ] ; then PATH=\"$HOME/.local/bin:$PATH\" fi export TZ=Asia/Singapore export NODE_ID=$(docker info -f '{{.Swarm.NodeID}}') export EMAIL=furyx@hotmail.com export DOMAIN=furyhawk.lol sudo nano /etc/fstab touch .credentials nano .credentials sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc sudo chmod a+r /etc/apt/keyrings/docker.asc echo \\ \"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \\ $(. /etc/os-release && echo \"$VERSION_CODENAME\") stable\" | \\ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin # sudo groupadd docker sudo usermod -aG docker $USER sudo systemctl enable docker.service sudo systemctl enable containerd.service systemctl list-units --type=service --state=active cd /etc/docker sudo touch daemon.json sudo nano daemon.json { \"log-driver\": \"json-file\", \"log-opts\": { \"max-size\": \"10m\", \"max-file\": \"3\" } } docker swarm join --token SWMTKN-1-xxx 192.168.50.114:2377 docker node update --availability drain node docker node update --availability Active node curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm chmod +x cephadm sudo ./cephadm add-repo --release octopus sudo ./cephadm install cephadm install ceph-common mkdir -p /etc/ceph sudo ./cephadm bootstrap --mon-ip 192.168.65.19 URL: https://debian.local:8443/ User: admin Password: 12345678 You can access the Ceph CLI with: sudo ./cephadm shell --fsid 8ad7deb6-265a-11ef-9e81-c2d0c41fc7e0 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring Please consider enabling telemetry to help improve Ceph: ceph telemetry on For more information see: https://docs.ceph.com/docs/master/mgr/telemetry/ alias ceph='./cephadm shell -- ceph' sudo ceph -v ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) sudo ceph orch host label add debian mon Added label mon to host debian sudo ceph orch apply mon debian Scheduled mon update... sudo ceph orch host ls ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-2 ssh-copy-id -f -i /etc/ceph/ceph.pub osd1 ssh-copy-id -f -i /etc/ceph/ceph.pub osd2 ssh-copy-id -f -i /etc/ceph/ceph.pub osd3 sudo ceph orch host add osd1 sudo ceph orch host add osd2 sudo ceph orch host add osd3 sudo ceph orch apply osd --all-available-devices sudo ceph status sudo find / -iname 'cephadm*' 2>/dev/null systemctl --user start docker-desktop sudo groupadd docker sudo usermod -aG docker $USER groups ${USER} curl -fsSL test.docker.com -o get-docker.sh && sh get-docker.sh sudo apt-get install libffi-dev libssl-dev sudo apt install python3-dev sudo apt-get install -y python3 python3-pip pip install \"cython<3.0.0\" wheel pip install \"pyyaml==5.4.1\" --no-build-isolation pip install docker-compose sudo systemctl enable docker sudo apt-get purge docker-ce sudo apt-get purge docker-ce-cli sudo rm -rf /var/lib/docker rootless sudo apt install -y dbus-user-session sudo apt install -y fuse-overlayfs sudo apt install -y fuse-overlayfs sudo systemctl disable --now docker.service docker.socket sudo rm -rf /var/lib/docker sudo apt-get purge docker-ce docker-ce-cli containerd.io sudo apt-get purge docker-ce docker-ce-cli containerd.io apt-get install -y uidmap curl -fsSL https://get.docker.com/rootless | sh nano .zshrc export PATH=/home/furyhawk/bin:$PATH export DOCKER_HOST=unix:///run/user/1000/docker.sock systemctl --user enable docker sudo loginctl enable-linger $(whoami) docker context use rootless sudo setcap cap_net_bind_service=ep $(which rootlesskit) systemctl --user restart docker #sudo apt-get install -y docker-ce-rootless-extras dockerd-rootless-setuptool.sh uninstall cd ~/bin rm -f containerd containerd-shim containerd-shim-runc-v2 ctr docker docker-init docker-proxy dockerd dockerd-rootless-setuptool.sh dockerd-rootless.sh rootlesskit rootlesskit-docker-proxy runc vpnkit netstat -ltup The Compose Specification {:.no_toc} ToC {:toc} Status of this document This document specifies the Compose file format used to define multi-containers applications. Distribution of this document is unlimited. Requirements and optional attributes The Compose specification includes properties designed to target a local OCI container runtime, exposing Linux kernel specific configuration options, but also some Windows container specific properties. It is also designed for cloud platform features related to resource placement on a cluster, replicated application distribution, and scalability. We acknowledge that no Compose implementation is expected to support all attributes, and that support for some properties is platform dependent and can only be confirmed at runtime. The definition of a versioned schema to control the supported properties in a Compose file, established by the docker-compose tool where the Compose file format was designed, doesn't offer any guarantee to the end-user that attributes will be actually implemented. The specification defines the expected configuration syntax and behavior. Unless noted, supporting any of these is optional. A Compose implementation to parse a Compose file using unsupported attributes should warn users. We recommend the following implementors to support those running modes: Default: warn the user about unsupported attributes, but ignore them Strict: warn the user about unsupported attributes and reject the Compose file Loose: ignore unsupported attributes AND unknown attributes (that were not defined by the spec by the time implementation was created) From this point onwards, references made to 'Compose' can be interpreted as 'a Compose implementation'. The Compose application model The Compose Specification lets you define a platform-agnostic container based application. Such an application is designed as a set of containers which have to both run together with adequate shared resources and communication channels. Computing components of an application are defined as services . A service is an abstract concept implemented on platforms by running the same container image, and configuration, one or more times. Services communicate with each other through networks . In the Compose Specification, a network is a platform capability abstraction to establish an IP route between containers within services connected together. Low-level, platform-specific networking options are grouped into the Network definition and may be partially implemented on some platforms. Services store and share persistent data into volumes . The Specification describes such a persistent data as a high-level filesystem mount with global options. Actual platform-specific implementation details are grouped into the volumes definition and may be partially implemented on some platforms. Some services require configuration data that is dependent on the runtime or platform. For this, the Specification defines a dedicated configs concept. From a service container point of view, configs are comparable to volumes, in that they are files mounted into the container. But the actual definition involves distinct platform resources and services, which are abstracted by this type. A secret is a specific flavor of configuration data for sensitive data that should not be exposed without security considerations. Secrets are made available to services as files mounted into their containers, but the platform-specific resources to provide sensitive data are specific enough to deserve a distinct concept and definition within the Compose specification. Note With volumes, configs and secrets you can have a simple declaration at the top-level and then add more platform-specific information at the service level. A project is an individual deployment of an application specification on a platform. A project's name, set with the top-level name attribute, is used to group resources together and isolate them from other applications or other installation of the same Compose specified application with distinct parameters. If you are creating resources on a platform, you must prefix resource names by project and set the label com.docker.compose.project . Compose offers a way for users to set a custom project name and override this name, so that the same compose.yaml file can be deployed twice on the same infrastructure, without changes, by just passing a distinct name. Project names must contain only lowercase letters, decimal digits, dashes, and underscores, and must begin with a lowercase letter or decimal digit. Illustrative example The following example illustrates the Compose Specification concepts outlined above. The example is non-normative. Consider an application split into a frontend web application and a backend service. The frontend is configured at runtime with an HTTP configuration file managed by infrastructure, providing an external domain name, and an HTTPS server certificate injected by the platform's secured secret store. The backend stores data in a persistent volume. Both services communicate with each other on an isolated back-tier network, while the frontend is also connected to a front-tier network and exposes port 443 for external usage. %%{ init: { 'flowchart': { 'curve': 'linear' } } }%% flowchart LR subgraph A[INFRASTRUCTURE] direction TB subgraph TOP[\" \"] subgraph B1[Frontend Service] fs[\"`**webapp**`\"] end style B1 fill:#ccd6e8, stroke-width:0px subgraph B2[Backend Service] bs[\"`**database**`\"] end style B2 fill:#ccd6e8, stroke-width:0px end style TOP fill:transparent, stroke-width:2px, stroke:#62affb, stroke-dasharray: 5 5 key[ro= read only\\nr+w = read write] style key fill:transparent, stroke-width:0px,text-align: left, size: 94px direction TB id2(Server\\nCertificate) id1(HTTP\\nConfiguration) id1 & id2 -.-|ro| B1 style id1 stroke:#000,stroke-width:1px,stroke-dasharray: 10 style id2 stroke:#000,stroke-width:1px,stroke-dasharray: 10 B2 ==r+w==> id3[(Persistent\\nVolume)] end style A fill:#eeeeee, stroke-width:0px direction LR id4[External\\nUser] ---id5(((443)))--->|Frontend\\nNetwork| B1 style id4 stroke:#000,stroke-width:2px B1 --Backend\\nNetwork--> B2 The example application is composed of the following parts: 2 services, backed by Docker images: webapp and database 1 secret (HTTPS certificate), injected into the frontend 1 configuration (HTTP), injected into the frontend 1 persistent volume, attached to the backend 2 networks services: frontend: image: example/webapp ports: - \"443:8043\" networks: - front-tier - back-tier configs: - httpd-config secrets: - server-certificate backend: image: example/database volumes: - db-data:/etc/data networks: - back-tier volumes: db-data: driver: flocker driver_opts: size: \"10GiB\" configs: httpd-config: external: true secrets: server-certificate: external: true networks: ## The presence of these objects is sufficient to define them front-tier: {} back-tier: {} This example illustrates the distinction between volumes, configs and secrets. While all of them are all exposed to service containers as mounted files or directories, only a volume can be configured for read+write access. Secrets and configs are read-only. The volume configuration allows you to select a volume driver and pass driver options to tweak volume management according to the actual infrastructure. Configs and secrets rely on platform services, and are declared external as they are not managed as part of the application lifecycle. Compose uses a platform-specific lookup mechanism to retrieve runtime values. Compose file The Compose file is a YAML file defining: - Version (Optional) - Services (Required) - Networks - Volumes - Configs - Secrets The default path for a Compose file is compose.yaml (preferred) or compose.yml that is placed in the working directory. Compose also supports docker-compose.yaml and docker-compose.yml for backwards compatibility of earlier versions. If both files exist, Compose prefers the canonical compose.yaml . You can use fragments and extensions to keep your Compose file efficient and easy to maintain. Multiple Compose files can be merged together to define the application model. The combination of YAML files are implemented by appending or overriding YAML elements based on the Compose file order you set. Simple attributes and maps get overridden by the highest order Compose file, lists get merged by appending. Relative paths are resolved based on the first Compose file's parent folder, whenever complimentary files being merged are hosted in other folders. As some Compose file elements can both be expressed as single strings or complex objects, merges apply to the expanded form. If you want to reuse other Compose files, or factor out parts of you application model into separate Compose files, you can also use include . This is useful if your Compose application is dependent on another application which is managed by a different team, or needs to be shared with others. Version and name top-level elements Version top-level element The top-level version property is defined by the Compose Specification for backward compatibility. It is only informative. Compose doesn't use version to select an exact schema to validate the Compose file, but prefers the most recent schema when it's implemented. Compose validates whether it can fully parse the Compose file. If some fields are unknown, typically because the Compose file was written with fields defined by a newer version of the Specification, you'll receive a warning message. Compose offers options to ignore unknown fields (as defined by \"loose\" mode). Name top-level element The top-level name property is defined by the Specification as the project name to be used if you don't set one explicitly. Compose offers a way for you to override this name, and sets a default project name to be used if the top-level name element is not set. Whenever a project name is defined by top-level name or by some custom mechanism, it is exposed for interpolation and environment variable resolution as COMPOSE_PROJECT_NAME services: foo: image: busybox environment: - COMPOSE_PROJECT_NAME command: echo \"I'm running ${COMPOSE_PROJECT_NAME}\" Services top-level element A service is an abstract definition of a computing resource within an application which can be scaled or replaced independently from other components. Services are backed by a set of containers, run by the platform according to replication requirements and placement constraints. As services are backed by containers, they are defined by a Docker image and set of runtime arguments. All containers within a service are identically created with these arguments. A Compose file must declare a services top-level element as a map whose keys are string representations of service names, and whose values are service definitions. A service definition contains the configuration that is applied to each service container. Each service may also include a build section, which defines how to create the Docker image for the service. Compose supports building docker images using this service definition. If not used, the build section is ignored and the Compose file is still considered valid. Build support is an optional aspect of the Compose Specification, and is described in detail in the Compose Build Specification documentation. Each service defines runtime constraints and requirements to run its containers. The deploy section groups these constraints and allows the platform to adjust the deployment strategy to best match containers' needs with available resources. Deploy support is an optional aspect of the Compose Specification, and is described in detail in the Compose Deploy Specification documentation. If not implemented the deploy section is ignored and the Compose file is still considered valid. attach When attach is defined and set to false Compose does not collect service logs, until you explicitly request it to. The default service configuration is attach: true . build build specifies the build configuration for creating a container image from source, as defined in the Compose Build Specification . blkio_config blkio_config defines a set of configuration options to set block IO limits for a service. services: foo: image: busybox blkio_config: weight: 300 weight_device: - path: /dev/sda weight: 400 device_read_bps: - path: /dev/sdb rate: '12mb' device_read_iops: - path: /dev/sdb rate: 120 device_write_bps: - path: /dev/sdb rate: '1024k' device_write_iops: - path: /dev/sdb rate: 30 device_read_bps, device_write_bps Set a limit in bytes per second for read / write operations on a given device. Each item in the list must have two keys: path : Defines the symbolic path to the affected device. rate : Either as an integer value representing the number of bytes or as a string expressing a byte value. device_read_iops, device_write_iops Set a limit in operations per second for read / write operations on a given device. Each item in the list must have two keys: path : Defines the symbolic path to the affected device. rate : As an integer value representing the permitted number of operations per second. weight Modify the proportion of bandwidth allocated to a service relative to other services. Takes an integer value between 10 and 1000, with 500 being the default. weight_device Fine-tune bandwidth allocation by device. Each item in the list must have two keys: path : Defines the symbolic path to the affected device. weight : An integer value between 10 and 1000. cpu_count cpu_count defines the number of usable CPUs for service container. cpu_percent cpu_percent defines the usable percentage of the available CPUs. cpu_shares cpu_shares defines, as integer value, a service container's relative CPU weight versus other containers. cpu_period cpu_period configures CPU CFS (Completely Fair Scheduler) period when a platform is based on Linux kernel. cpu_quota cpu_quota configures CPU CFS (Completely Fair Scheduler) quota when a platform is based on Linux kernel. cpu_rt_runtime cpu_rt_runtime configures CPU allocation parameters for platforms with support for realtime scheduler. It can be either an integer value using microseconds as unit or a duration . cpu_rt_runtime: '400ms' cpu_rt_runtime: 95000` cpu_rt_period cpu_rt_period configures CPU allocation parameters for platforms with support for realtime scheduler. It can be either an integer value using microseconds as unit or a duration . cpu_rt_period: '1400us' cpu_rt_period: 11000` cpus DEPRECATED: use deploy.limits.cpus cpus define the number of (potentially virtual) CPUs to allocate to service containers. This is a fractional number. 0.000 means no limit. cpuset cpuset defines the explicit CPUs in which to allow execution. Can be a range 0-3 or a list 0,1 cap_add cap_add specifies additional container capabilities as strings. cap_add: - ALL cap_drop cap_drop specifies container capabilities to drop as strings. cap_drop: - NET_ADMIN - SYS_ADMIN cgroup cgroup specifies the cgroup namespace to join. When unset, it is the container runtime's decision to select which cgroup namespace to use, if supported. host : Runs the container in the Container runtime cgroup namespace. private : Runs the container in its own private cgroup namespace. cgroup_parent cgroup_parent specifies an optional parent cgroup for the container. cgroup_parent: m-executor-abcd command command overrides the default command declared by the container image, for example by Dockerfile's CMD . command: bundle exec thin -p 3000 The value can also be a list, in a manner similar to Dockerfile : command: [ \"bundle\", \"exec\", \"thin\", \"-p\", \"3000\" ] If the value is null , the default command from the image is used. If the value is [] (empty list) or '' (empty string), the default command declared by the image is ignored, i.e. overridden to be empty. configs Configs allow services to adapt their behaviour without the need to rebuild a Docker image. Services can only access configs when explicitly granted by the configs attribute. Two different syntax variants are supported. Compose reports an error if config doesn't exist on the platform or isn't defined in the configs top-level element in the Compose file. There are two syntaxes defined for configs. To remain compliant to this specification, an implementation must support both syntaxes. Implementations must allow use of both short and long syntaxes within the same document. You can grant a service access to multiple configs, and you can mix long and short syntax. Short syntax The short syntax variant only specifies the config name. This grants the container access to the config and mounts it as files into a service\u2019s container\u2019s filesystem. The location of the mount point within the container defaults to / in Linux containers, and C:\\ in Windows containers. The following example uses the short syntax to grant the redis service access to the my_config and my_other_config configs. The value of my_config is set to the contents of the file ./my_config.txt , and my_other_config is defined as an external resource, which means that it has already been defined in the platform. If the external config does not exist, the deployment fails. services: redis: image: redis:latest configs: - my_config - my_other_config configs: my_config: file: ./my_config.txt my_other_config: external: true Long syntax The long syntax provides more granularity in how the config is created within the service's task containers. source : The name of the config as it exists in the platform. target : The path and name of the file to be mounted in the service's task containers. Defaults to / if not specified. uid and gid : The numeric UID or GID that owns the mounted config file within the service's task containers. Default value when not specified is USER running container. mode : The permissions for the file that is mounted within the service's task containers, in octal notation. Default value is world-readable ( 0444 ). Writable bit must be ignored. The executable bit can be set. The following example sets the name of my_config to redis_config within the container, sets the mode to 0440 (group-readable) and sets the user and group to 103 . The redis service does not have access to the my_other_config config. services: redis: image: redis:latest configs: - source: my_config target: /redis_config uid: \"103\" gid: \"103\" mode: 0440 configs: my_config: external: true my_other_config: external: true container_name container_name is a string that specifies a custom container name, rather than a name generated by default. container_name: my-web-container Compose does not scale a service beyond one container if the Compose file specifies a container_name . Attempting to do so results in an error. container_name follows the regex format of [a-zA-Z0-9][a-zA-Z0-9_.-]+ credential_spec credential_spec configures the credential spec for a managed service account. If you have services that use Windows containers, you can use file: and registry: protocols for credential_spec . Compose also supports additional protocols for custom use-cases. The credential_spec must be in the format file:// or registry:// . credential_spec: file: my-credential-spec.json When using registry: , the credential spec is read from the Windows registry on the daemon's host. A registry value with the given name must be located in: HKLM\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Virtualization\\Containers\\CredentialSpecs The following example loads the credential spec from a value named my-credential-spec in the registry: credential_spec: registry: my-credential-spec Example gMSA configuration When configuring a gMSA credential spec for a service, you only need to specify a credential spec with config , as shown in the following example: services: myservice: image: myimage:latest credential_spec: config: my_credential_spec configs: my_credentials_spec: file: ./my-credential-spec.json| depends_on depends_on expresses startup and shutdown dependencies between services. Short syntax The short syntax variant only specifies service names of the dependencies. Service dependencies cause the following behaviors: Compose creates services in dependency order. In the following example, db and redis are created before web . Compose removes services in dependency order. In the following example, web is removed before db and redis . Simple example: services: web: build: . depends_on: - db - redis redis: image: redis db: image: postgres Compose guarantees dependency services have been started before starting a dependent service. Compose waits for dependency services to be \"ready\" before starting a dependent service. Long syntax The long form syntax enables the configuration of additional fields that can't be expressed in the short form. restart : When set to true Compose restarts this service after it updates the dependency service. This applies to an explicit restart controlled by a Compose operation, and excludes automated restart by the container runtime after the container dies. condition : Sets the condition under which dependency is considered satisfied service_started : An equivalent of the short syntax described above service_healthy : Specifies that a dependency is expected to be \"healthy\" (as indicated by healthcheck ) before starting a dependent service. service_completed_successfully : Specifies that a dependency is expected to run to successful completion before starting a dependent service. required : When set to false Compose only warns you when the dependency service isn't started or available. If it's not defined the default value of required is true . Service dependencies cause the following behaviors: Compose creates services in dependency order. In the following example, db and redis are created before web . Compose waits for healthchecks to pass on dependencies marked with service_healthy . In the following example, db is expected to be \"healthy\" before web is created. Compose removes services in dependency order. In the following example, web is removed before db and redis . services: web: build: . depends_on: db: condition: service_healthy restart: true redis: condition: service_started redis: image: redis db: image: postgres Compose guarantees dependency services are started before starting a dependent service. Compose guarantees dependency services marked with service_healthy are \"healthy\" before starting a dependent service. deploy deploy specifies the configuration for the deployment and lifecycle of services, as defined in the Compose Deploy Specification . develop develop specifies the development configuration for maintaining a container in sync with source, as defined in the Development Section . device_cgroup_rules device_cgroup_rules defines a list of device cgroup rules for this container. The format is the same format the Linux kernel specifies in the Control Groups Device Whitelist Controller . device_cgroup_rules: - 'c 1:3 mr' - 'a 7:* rmw' devices devices defines a list of device mappings for created containers in the form of HOST_PATH:CONTAINER_PATH[:CGROUP_PERMISSIONS] . devices: - \"/dev/ttyUSB0:/dev/ttyUSB0\" - \"/dev/sda:/dev/xvda:rwm\" dns dns defines custom DNS servers to set on the container network interface configuration. It can be a single value or a list. dns: 8.8.8.8 dns: - 8.8.8.8 - 9.9.9.9 dns_opt dns_opt list custom DNS options to be passed to the container\u2019s DNS resolver ( /etc/resolv.conf file on Linux). dns_opt: - use-vc - no-tld-query dns_search dns_search defines custom DNS search domains to set on container network interface configuration. It can be a single value or a list. dns_search: example.com dns_search: - dc1.example.com - dc2.example.com domainname domainname declares a custom domain name to use for the service container. It must be a valid RFC 1123 hostname. entrypoint entrypoint declares the default entrypoint for the service container. This overrides the ENTRYPOINT instruction from the service's Dockerfile. If entrypoint is non-null, Compose ignores any default command from the image, for example the CMD instruction in the Dockerfile. See also command to set or override the default command to be executed by the entrypoint process. In its short form, the value can be defined as a string: entrypoint: /code/entrypoint.sh Alternatively, the value can also be a list, in a manner similar to the Dockerfile : entrypoint: - php - -d - zend_extension=/usr/local/lib/php/extensions/no-debug-non-zts-20100525/xdebug.so - -d - memory_limit=-1 - vendor/bin/phpunit If the value is null , the default entrypoint from the image is used. If the value is [] (empty list) or '' (empty string), the default entrypoint declared by the image is ignored, i.e. overridden to be empty. env_file env_file adds environment variables to the container based on the file content. env_file: .env env_file can also be a list. The files in the list are processed from the top down. For the same variable specified in two env files, the value from the last file in the list stands. env_file: - ./a.env - ./b.env List elements can also be declared as a mapping, which then lets you set an additional attribute required . This defaults to true . When required is set to false and the .env file is missing, Compose silently ignores the entry. env_file: - path: ./default.env required: true ## default - path: ./override.env required: false Relative path are resolved from the Compose file's parent folder. As absolute paths prevent the Compose file from being portable, Compose warns you when such a path is used to set env_file . Environment variables declared in the environment section override these values. This holds true even if those values are empty or undefined. Env_file format Each line in an .env file must be in VAR[=[VAL]] format. The following syntax rules apply: Lines beginning with # are processed as comments and ignored. Blank lines are ignored. Unquoted and double-quoted ( \" ) values have Interpolation applied. Each line represents a key-value pair. Values can optionally be quoted. VAR=VAL -> VAL VAR=\"VAL\" -> VAL VAR='VAL' -> VAL Inline comments for unquoted values must be preceded with a space. VAR=VAL ## comment -> VAL VAR=VAL## not a comment -> VAL## not a comment Inline comments for quoted values must follow the closing quote. VAR=\"VAL ## not a comment\" -> VAL ## not a comment VAR=\"VAL\" ## comment -> VAL Single-quoted ( ' ) values are used literally. VAR='$OTHER' -> $OTHER VAR='${OTHER}' -> ${OTHER} Quotes can be escaped with \\ . VAR='Let\\'s go!' -> Let's go! VAR=\"{\\\"hello\\\": \\\"json\\\"}\" -> {\"hello\": \"json\"} Common shell escape sequences including \\n , \\r , \\t , and \\\\ are supported in double-quoted values. VAR=\"some\\tvalue\" -> some value VAR='some\\tvalue' -> some\\tvalue VAR=some\\tvalue -> some\\tvalue VAL may be omitted, in such cases the variable value is an empty string. =VAL may be omitted, in such cases the variable is unset. ## Set Rails/Rack environment RACK_ENV=development VAR=\"quoted\" environment environment defines environment variables set in the container. environment can use either an array or a map. Any boolean values; true, false, yes, no, should be enclosed in quotes to ensure they are not converted to True or False by the YAML parser. Environment variables can be declared by a single key (no value to equals sign). In this case Compose relies on you to resolve the value. If the value is not resolved, the variable is unset and is removed from the service container environment. Map syntax: environment: RACK_ENV: development SHOW: \"true\" USER_INPUT: Array syntax: environment: - RACK_ENV=development - SHOW=true - USER_INPUT When both env_file and environment are set for a service, values set by environment have precedence. expose expose defines the (incoming) port or a range of ports that Compose exposes from the container. These ports must be accessible to linked services and should not be published to the host machine. Only the internal container ports can be specified. Syntax is /[] or /[] for a port range. When not explicitly set, tcp protocol is used. expose: - \"3000\" - \"8000\" - \"8080-8085/tcp Note If the Dockerfile for the image already exposes ports, it is visible to other containers on the network even if expose is not set in your Compose file. extends extends lets you share common configurations among different files, or even different projects entirely. With extends you can define a common set of service options in one place and refer to it from anywhere. You can refer to another Compose file and select a service you want to also use in your own application, with the ability to override some attributes for your own needs. You can use extends on any service together with other configuration keys. The extends value must be a mapping defined with a required service and an optional file key. extends: file: common.yml service: webapp service : Defines the name of the service being referenced as a base, for example web or database . file : The location of a Compose configuration file defining that service. Restrictions The following restrictions apply to the service being referenced: Services that have dependencies on other services cannot be used as a base. Therefore, any key that introduces a dependency on another service is incompatible with extends . The non-exhaustive list of such keys is: links , volumes_from , container mode (in ipc , pid , network_mode and net ), service mode (in ipc , pid and network_mode ), depends_on . Services cannot have circular references with extends . Compose returns an error in all of these cases. Finding referenced service file value can be: Not present. This indicates that another service within the same Compose file is being referenced. File path, which can be either: Relative path. This path is considered as relative to the location of the main Compose file. Absolute path. A service denoted by service must be present in the identified referenced Compose file. Compose returns an error if: The service denoted by service is not found. The Compose file denoted by file is not found. Merging service definitions Two service definitions, the main one in the current Compose file and the referenced one specified by extends , are merged in the following way: Mappings: Keys in mappings of the main service definition override keys in mappings of the referenced service definition. Keys that aren't overridden are included as is. Sequences: Items are combined together into a new sequence. The order of elements is preserved with the referenced items coming first and main items after. Scalars: Keys in the main service definition take precedence over keys in the referenced one. Mappings The following keys should be treated as mappings: annotations , build.args , build.labels , build.extra_hosts , deploy.labels , deploy.update_config , deploy.rollback_config , deploy.restart_policy , deploy.resources.limits , environment , healthcheck , labels , logging.options , sysctls , storage_opt , extra_hosts , ulimits . One exception that applies to healthcheck is that the main mapping cannot specify disable: true unless the referenced mapping also specifies disable: true . Compose returns an error in this case. For example, the input below: services: common: image: busybox environment: TZ: utc PORT: 80 cli: extends: service: common environment: PORT: 8080 Produces the following configuration for the cli service. The same output is produced if array syntax is used. environment: PORT: 8080 TZ: utc image: busybox Items under blkio_config.device_read_bps , blkio_config.device_read_iops , blkio_config.device_write_bps , blkio_config.device_write_iops , devices and volumes are also treated as mappings where key is the target path inside the container. For example, the input below: services: common: image: busybox volumes: - common-volume:/var/lib/backup/data:rw cli: extends: service: common volumes: - cli-volume:/var/lib/backup/data:ro Produces the following configuration for the cli service. Note that the mounted path now points to the new volume name and ro flag was applied. image: busybox volumes: - cli-volume:/var/lib/backup/data:ro If the referenced service definition contains extends mapping, the items under it are simply copied into the new merged definition. The merging process is then kicked off again until no extends keys are remaining. For example, the input below: services: base: image: busybox user: root common: image: busybox extends: service: base cli: extends: service: common Produces the following configuration for the cli service. Here, cli services gets user key from common service, which in turn gets this key from base service. image: busybox user: root Sequences The following keys should be treated as sequences: cap_add , cap_drop , configs , deploy.placement.constraints , deploy.placement.preferences , deploy.reservations.generic_resources , device_cgroup_rules , expose , external_links , ports , secrets , security_opt . Any duplicates resulting from the merge are removed so that the sequence only contains unique elements. For example, the input below: services: common: image: busybox security_opt: - label:role:ROLE cli: extends: service: common security_opt: - label:user:USER Produces the following configuration for the cli service. image: busybox security_opt: - label:role:ROLE - label:user:USER In case list syntax is used, the following keys should also be treated as sequences: dns , dns_search , env_file , tmpfs . Unlike sequence fields mentioned above, duplicates resulting from the merge are not removed. Scalars Any other allowed keys in the service definition should be treated as scalars. annotations annotations defines annotations for the container. annotations can use either an array or a map. annotations: com.example.foo: bar annotations: - com.example.foo=bar external_links external_links link service containers to services managed outside of your Compose application. external_links define the name of an existing service to retrieve using the platform lookup mechanism. An alias of the form SERVICE:ALIAS can be specified. external_links: - redis - database:mysql - database:postgresql extra_hosts extra_hosts adds hostname mappings to the container network interface configuration ( /etc/hosts for Linux). Short syntax Short syntax uses plain strings in a list. Values must set hostname and IP address for additional hosts in the form of HOSTNAME=IP . extra_hosts: - \"somehost=162.242.195.82\" - \"otherhost=50.31.209.229\" - \"myhostv6=::1\" IPv6 addresses can be enclosed in square brackets, for example: extra_hosts: - \"myhostv6=[::1]\" The separator = is preferred, but : can also be used. For example: extra_hosts: - \"somehost:162.242.195.82\" - \"myhostv6:::1\" Long syntax Alternatively, extra_hosts can be set as a mapping between hostname(s) and IP(s) extra_hosts: somehost: \"162.242.195.82\" otherhost: \"50.31.209.229\" myhostv6: \"::1\" Compose creates a matching entry with the IP address and hostname in the container's network configuration, which means for Linux /etc/hosts get extra lines: 162.242.195.82 somehost 50.31.209.229 otherhost ::1 myhostv6 group_add group_add specifies additional groups, by name or number, which the user inside the container must be a member of. An example of where this is useful is when multiple containers (running as different users) need to all read or write the same file on a shared volume. That file can be owned by a group shared by all the containers, and specified in group_add . services: myservice: image: alpine group_add: - mail Running id inside the created container must show that the user belongs to the mail group, which would not have been the case if group_add were not declared. healthcheck healthcheck declares a check that's run to determine whether or not the service containers are \"healthy\". It works in the same way, and has the same default values, as the HEALTHCHECK Dockerfile instruction set by the service's Docker image. Your Compose file can override the values set in the Dockerfile. healthcheck: test: [\"CMD\", \"curl\", \"-f\", \"http://localhost\"] interval: 1m30s timeout: 10s retries: 3 start_period: 40s start_interval: 5s interval , timeout , start_period , and start_interval are specified as durations . test defines the command Compose runs to check container health. It can be either a string or a list. If it's a list, the first item must be either NONE , CMD or CMD-SHELL . If it's a string, it's equivalent to specifying CMD-SHELL followed by that string. ## Hit the local web app test: [\"CMD\", \"curl\", \"-f\", \"http://localhost\"] Using CMD-SHELL runs the command configured as a string using the container's default shell ( /bin/sh for Linux). Both forms below are equivalent: test: [\"CMD-SHELL\", \"curl -f http://localhost || exit 1\"] test: curl -f https://localhost || exit 1 NONE disables the healthcheck, and is mostly useful to disable the Healthcheck Dockerfile instruction set by the service's Docker image. Alternatively, the healthcheck set by the image can be disabled by setting disable: true : healthcheck: disable: true hostname hostname declares a custom host name to use for the service container. It must be a valid RFC 1123 hostname. image image specifies the image to start the container from. image must follow the Open Container Specification addressable image format , as [/][/][:|@] . image: redis image: redis:5 image: redis@sha256:0ed5d5928d4737458944eb604cc8509e245c3e19d02ad83935398bc4b991aac7 image: library/redis image: docker.io/library/redis image: my_private.registry:5000/redis If the image does not exist on the platform, Compose attempts to pull it based on the pull_policy . If you are also using the Compose Build Specification , there are alternative options for controlling the precedence of pull over building the image from source, however pulling the image is the default behavior. image may be omitted from a Compose file as long as a build section is declared. If you are not using the Compose Build Specification, Compose won't work if image is missing from the Compose file. init init runs an init process (PID 1) inside the container that forwards signals and reaps processes. Set this option to true to enable this feature for the service. services: web: image: alpine:latest init: true The init binary that is used is platform specific. ipc ipc configures the IPC isolation mode set by the service container. Available values are platform specific, but Compose defines specific values which must be implemented as described if supported: shareable : Gives the container its own private IPC namespace, with a possibility to share it with other containers. service:{name} : Makes the container join another container's ( shareable ) IPC namespace. ipc: \"shareable\" ipc: \"service:[service name]\" uts uts configures the UTS namespace mode set for the service container. When unspecified it is the runtime's decision to assign a UTS namespace, if supported. Available values are: 'host' : Results in the container using the same UTS namespace as the host. uts: \"host\" isolation isolation specifies a container\u2019s isolation technology. Supported values are platform specific. labels labels add metadata to containers. You can use either an array or a map. It's recommended that you use reverse-DNS notation to prevent your labels from conflicting with those used by other software. labels: com.example.description: \"Accounting webapp\" com.example.department: \"Finance\" com.example.label-with-empty-value: \"\" labels: - \"com.example.description=Accounting webapp\" - \"com.example.department=Finance\" - \"com.example.label-with-empty-value\" Compose creates containers with canonical labels: com.docker.compose.project set on all resources created by Compose to the user project name com.docker.compose.service set on service containers with service name as defined in the Compose file The com.docker.compose label prefix is reserved. Specifying labels with this prefix in the Compose file results in a runtime error. links Note Availability of the links attribute is implementation specific. links defines a network link to containers in another service. Either specify both the service name and a link alias ( SERVICE:ALIAS ), or just the service name. web: links: - db - db:database - redis Containers for the linked service are reachable at a hostname identical to the alias, or the service name if no alias is specified. Links are not required to enable services to communicate. When no specific network configuration is set, any service is able to reach any other service at that service\u2019s name on the default network. If services do declare networks they are attached to, links does not override the network configuration and services not attached to a shared network are not be able to communicate. Compose doesn't warn you about a configuration mismatch. Links also express implicit dependency between services in the same way as depends_on , so they determine the order of service startup. logging logging defines the logging configuration for the service. logging: driver: syslog options: syslog-address: \"tcp://192.168.0.42:123\" The driver name specifies a logging driver for the service's containers. The default and available values are platform specific. Driver specific options can be set with options as key-value pairs. network_mode network_mode sets a service container's network mode. Available values are platform specific, but Compose defines specific values which must be implemented as described if supported: none : Turns off all container networking. host : Gives the container raw access to the host's network interface. service:{name} : Gives the containers access to the specified service only. network_mode: \"host\" network_mode: \"none\" network_mode: \"service:[service name]\" When set, the networks attribute is not allowed and Compose rejects any Compose file containing both attributes. networks networks defines the networks that service containers are attached to, referencing entries under the top-level networks key . services: some-service: networks: - some-network - other-network aliases aliases declares alternative hostnames for the service on the network. Other containers on the same network can use either the service name or an alias to connect to one of the service's containers. Since aliases are network-scoped, the same service can have different aliases on different networks. Note A network-wide alias can be shared by multiple containers, and even by multiple services. If it is, then exactly which container the name resolves to is not guaranteed. services: some-service: networks: some-network: aliases: - alias1 - alias3 other-network: aliases: - alias2 In the following example, service frontend is able to reach the backend service at the hostname backend or database on the back-tier network. The service monitoring is able to reach same backend service at backend or mysql on the admin network. services: frontend: image: example/webapp networks: - front-tier - back-tier monitoring: image: example/monitoring networks: - admin backend: image: example/backend networks: back-tier: aliases: - database admin: aliases: - mysql networks: front-tier: back-tier: admin: ipv4_address, ipv6_address Specify a static IP address for a service container when joining the network. The corresponding network configuration in the top-level networks section must have an ipam attribute with subnet configurations covering each static address. services: frontend: image: example/webapp networks: front-tier: ipv4_address: 172.16.238.10 ipv6_address: 2001:3984:3989::10 networks: front-tier: ipam: driver: default config: - subnet: \"172.16.238.0/24\" - subnet: \"2001:3984:3989::/64\" link_local_ips link_local_ips specifies a list of link-local IPs. Link-local IPs are special IPs which belong to a well known subnet and are purely managed by the operator, usually dependent on the architecture where they are deployed. Implementation is platform specific. Example: services: app: image: busybox command: top networks: app_net: link_local_ips: - 57.123.22.11 - 57.123.22.13 networks: app_net: driver: bridge mac_address mac_address sets the MAC address used by the service container when connecting to this particular network. priority priority indicates in which order Compose connects the service\u2019s containers to its networks. If unspecified, the default value is 0. In the following example, the app service connects to app_net_1 first as it has the highest priority. It then connects to app_net_3 , then app_net_2 , which uses the default priority value of 0. services: app: image: busybox command: top networks: app_net_1: priority: 1000 app_net_2: app_net_3: priority: 100 networks: app_net_1: app_net_2: app_net_3: mac_address mac_address sets a MAC address for the service container. Note Container runtimes might reject this value (ie. Docker Engine >= v25.0). In that case, you should use networks.mac_address instead. mem_limit DEPRECATED: use deploy.limits.memory mem_reservation DEPRECATED: use deploy.reservations.memory mem_swappiness mem_swappiness defines as a percentage, a value between 0 and 100, for the host kernel to swap out anonymous memory pages used by a container. 0 : Turns off anonymous page swapping. 100 : Sets all anonymous pages as swappable. The default value is platform specific. memswap_limit memswap_limit defines the amount of memory the container is allowed to swap to disk. This is a modifier attribute that only has meaning if memory is also set. Using swap lets the container write excess memory requirements to disk when the container has exhausted all the memory that is available to it. There is a performance penalty for applications that swap memory to disk often. If memswap_limit is set to a positive integer, then both memory and memswap_limit must be set. memswap_limit represents the total amount of memory and swap that can be used, and memory controls the amount used by non-swap memory. So if memory =\"300m\" and memswap_limit =\"1g\", the container can use 300m of memory and 700m (1g - 300m) swap. If memswap_limit is set to 0, the setting is ignored, and the value is treated as unset. If memswap_limit is set to the same value as memory , and memory is set to a positive integer, the container does not have access to swap. If memswap_limit is unset, and memory is set, the container can use as much swap as the memory setting, if the host container has swap memory configured. For instance, if memory =\"300m\" and memswap_limit is not set, the container can use 600m in total of memory and swap. If memswap_limit is explicitly set to -1, the container is allowed to use unlimited swap, up to the amount available on the host system. oom_kill_disable If oom_kill_disable is set, Compose configures the platform so it won't kill the container in case of memory starvation. oom_score_adj oom_score_adj tunes the preference for containers to be killed by platform in case of memory starvation. Value must be within [-1000,1000] range. pid pid sets the PID mode for container created by Compose. Supported values are platform specific. pids_limit DEPRECATED: use deploy.resources.limits.pids pids_limit tunes a container\u2019s PIDs limit. Set to -1 for unlimited PIDs. pids_limit: 10 platform platform defines the target platform the containers for the service run on. It uses the os[/arch[/variant]] syntax. The values of os , arch , and variant must conform to the convention used by the OCI Image Spec . Compose uses this attribute to determine which version of the image is pulled and/or on which platform the service\u2019s build is performed. platform: darwin platform: windows/amd64 platform: linux/arm64/v8 ports Exposes container ports. Note Port mapping must not be used with network_mode: host otherwise a runtime error occurs. Short syntax The short syntax is a colon-separated string to set the host IP, host port, and container port in the form: [HOST:]CONTAINER[/PROTOCOL] where: HOST is [IP:](port | range) CONTAINER is port | range PROTOCOL to restrict port to specified protocol. tcp and udp values are defined by the Specification, Compose offers support for platform-specific protocol names. If host IP is not set, it binds to all network interfaces. Ports can be either a single value or a range. Host and container must use equivalent ranges. Either specify both ports ( HOST:CONTAINER ), or just the container port. In the latter case, the container runtime automatically allocates any unassigned port of the host. HOST:CONTAINER should always be specified as a (quoted) string, to avoid conflicts with yaml base-60 float . Examples: ports: - \"3000\" - \"3000-3005\" - \"8000:8000\" - \"9090-9091:8080-8081\" - \"49100:22\" - \"8000-9000:80\" - \"127.0.0.1:8001:8001\" - \"127.0.0.1:5000-5010:5000-5010\" - \"6060:6060/udp\" Note If Host IP mapping is not supported by a container engine, Compose rejects the Compose file and ignores the specified host IP. Long syntax The long form syntax allows the configuration of additional fields that can't be expressed in the short form. target : The container port published : The publicly exposed port. It is defined as a string and can be set as a range using syntax start-end . It means the actual port is assigned a remaining available port, within the set range. host_ip : The Host IP mapping, unspecified means all network interfaces ( 0.0.0.0 ). protocol : The port protocol ( tcp or udp ). Defaults to tcp . mode : host : For publishing a host port on each node, or ingress for a port to be load balanced. Defaults to ingress . name : A human-readable name for the port, used to document it's usage within the service ports: - name: http target: 80 host_ip: 127.0.0.1 published: \"8080\" protocol: tcp mode: host - name: https target: 443 host_ip: 127.0.0.1 published: \"8083-9000\" protocol: tcp mode: host privileged privileged configures the service container to run with elevated privileges. Support and actual impacts are platform specific. profiles profiles defines a list of named profiles for the service to be enabled under. If unassigned, the service is always started but if assigned, it is only started if the profile is activated. If present, profiles follow the regex format of [a-zA-Z0-9][a-zA-Z0-9_.-]+ . services: frontend: image: frontend profiles: [\"frontend\"] phpmyadmin: image: phpmyadmin depends_on: - db profiles: - debug pull_policy pull_policy defines the decisions Compose makes when it starts to pull images. Possible values are: always : Compose always pulls the image from the registry. never : Compose doesn't pull the image from a registry and relies on the platform cached image. If there is no cached image, a failure is reported. missing : Compose pulls the image only if it's not available in the platform cache. This is the default option if you are not also using the Compose Build Specification . if_not_present is considered an alias for this value for backward compatibility. build : Compose builds the image. Compose rebuilds the image if it's already present. read_only read_only configures the service container to be created with a read-only filesystem. restart restart defines the policy that the platform applies on container termination. no : The default restart policy. It does not restart the container under any circumstances. always : The policy always restarts the container until its removal. on-failure : The policy restarts the container if the exit code indicates an error. unless-stopped : The policy restarts the container irrespective of the exit code but stops restarting when the service is stopped or removed. restart: \"no\" restart: always restart: on-failure restart: unless-stopped runtime runtime specifies which runtime to use for the service\u2019s containers. The value of runtime is specific to the implementation. For example, runtime can be the name of an implementation of OCI Runtime Spec , such as \"runc\". web: image: busybox:latest command: true runtime: runc scale scale specifies the default number of containers to deploy for this service. When both are set, scale must be consistent with the replicas attribute in the Deploy Specification . secrets secrets grants access to sensitive data defined by secrets on a per-service basis. Two different syntax variants are supported; the short syntax and the long syntax. Compose reports an error if the secret doesn't exist on the platform or isn't defined in the secrets section of the Compose file. Services can be granted access to multiple secrets. Long and short syntax for secrets may be used in the same Compose file. Defining a secret in the top-level secrets must not imply granting any service access to it. Such grant must be explicit within service specification as secrets service element. Short syntax The short syntax variant only specifies the secret name. This grants the container access to the secret and mounts it as read-only to /run/secrets/ within the container. The source name and destination mountpoint are both set to the secret name. The following example uses the short syntax to grant the frontend service access to the server-certificate secret. The value of server-certificate is set to the contents of the file ./server.cert . services: frontend: image: example/webapp secrets: - server-certificate secrets: server-certificate: file: ./server.cert Long syntax The long syntax provides more granularity in how the secret is created within the service's containers. source : The name of the secret as it exists on the platform. target : The name of the file to be mounted in /run/secrets/ in the service's task container, or absolute path of the file if an alternate location is required. Defaults to source if not specified. uid and gid : The numeric UID or GID that owns the file within /run/secrets/ in the service's task containers. Default value is USER running container. mode : The permissions for the file to be mounted in /run/secrets/ in the service's task containers, in octal notation. The default value is world-readable permissions (mode 0444 ). The writable bit must be ignored if set. The executable bit may be set. Note that the uid , gid , and mode attributes are implementation specific. The following example sets the name of the server-certificate secret file to server.crt within the container, sets the mode to 0440 (group-readable), and sets the user and group to 103 . The value of server-certificate secret is provided by the platform through a lookup and the secret's lifecycle is not directly managed by Compose. services: frontend: image: example/webapp secrets: - source: server-certificate target: server.cert uid: \"103\" gid: \"103\" mode: 0440 secrets: server-certificate: external: true security_opt security_opt overrides the default labeling scheme for each container. security_opt: - label:user:USER - label:role:ROLE For further default labeling schemes you can override, see Security configuration . shm_size shm_size configures the size of the shared memory ( /dev/shm partition on Linux) allowed by the service container. It's specified as a byte value . stdin_open stdin_open configures a service containers to run with an allocated stdin. stop_grace_period stop_grace_period specifies how long Compose must wait when attempting to stop a container if it doesn't handle SIGTERM (or whichever stop signal has been specified with stop_signal ), before sending SIGKILL. It's specified as a duration . stop_grace_period: 1s stop_grace_period: 1m30s Default value is 10 seconds for the container to exit before sending SIGKILL. stop_signal stop_signal defines the signal that Compose uses to stop the service containers. If unset containers are stopped by Compose by sending SIGTERM . stop_signal: SIGUSR1 storage_opt storage_opt defines storage driver options for a service. storage_opt: size: '1G' sysctls sysctls defines kernel parameters to set in the container. sysctls can use either an array or a map. sysctls: net.core.somaxconn: 1024 net.ipv4.tcp_syncookies: 0 sysctls: - net.core.somaxconn=1024 - net.ipv4.tcp_syncookies=0 You can only use sysctls that are namespaced in the kernel. Docker does not support changing sysctls inside a container that also modify the host system. For an overview of supported sysctls, refer to configure namespaced kernel parameters (sysctls) at runtime . tmpfs tmpfs mounts a temporary file system inside the container. It can be a single value or a list. tmpfs: /run tmpfs: - /run - /tmp tty tty configures service container to run with a TTY. ulimits ulimits overrides the default ulimits for a container. It's specified either as an integer for a single limit or as mapping for soft/hard limits. ulimits: nproc: 65535 nofile: soft: 20000 hard: 40000 user user overrides the user used to run the container process. The default is set by the image (i.e. Dockerfile USER ). If it's not set, then root . userns_mode userns_mode sets the user namespace for the service. Supported values are platform specific and may depend on platform configuration. userns_mode: \"host\" volumes volumes define mount host paths or named volumes that are accessible by service containers. You can use volumes to define multiple types of mounts; volume , bind , tmpfs , or npipe . If the mount is a host path and is only used by a single service, it can be declared as part of the service definition. To reuse a volume across multiple services, a named volume must be declared in the top-level volumes key . The following example shows a named volume ( db-data ) being used by the backend service, and a bind mount defined for a single service. services: backend: image: example/backend volumes: - type: volume source: db-data target: /data volume: nocopy: true - type: bind source: /var/run/postgres/postgres.sock target: /var/run/postgres/postgres.sock volumes: db-data: Short syntax The short syntax uses a single string with colon-separated values to specify a volume mount ( VOLUME:CONTAINER_PATH ), or an access mode ( VOLUME:CONTAINER_PATH:ACCESS_MODE ). VOLUME : Can be either a host path on the platform hosting containers (bind mount) or a volume name. CONTAINER_PATH : The path in the container where the volume is mounted. ACCESS_MODE : A comma-separated , list of options: rw : Read and write access. This is the default if none is specified. ro : Read-only access. z : SELinux option indicating that the bind mount host content is shared among multiple containers. Z : SELinux option indicating that the bind mount host content is private and unshared for other containers. Note The SELinux re-labeling bind mount option is ignored on platforms without SELinux. Note Relative host paths are only supported by Compose that deploy to a local container runtime. This is because the relative path is resolved from the Compose file\u2019s parent directory which is only applicable in the local case. When Compose deploys to a non-local platform it rejects Compose files which use relative host paths with an error. To avoid ambiguities with named volumes, relative paths should always begin with . or .. . Long syntax The long form syntax allows the configuration of additional fields that can't be expressed in the short form. type : The mount type. Either volume , bind , tmpfs , npipe , or cluster source : The source of the mount, a path on the host for a bind mount, or the name of a volume defined in the top-level volumes key . Not applicable for a tmpfs mount. target : The path in the container where the volume is mounted. read_only : Flag to set the volume as read-only. bind : Used to configure additional bind options: propagation : The propagation mode used for the bind. create_host_path : Creates a directory at the source path on host if there is nothing present. Compose does nothing if there is something present at the path. This is automatically implied by short syntax for backward compatibility with docker-compose legacy. selinux : The SELinux re-labeling option z (shared) or Z (private) volume : Configures additional volume options: nocopy : Flag to disable copying of data from a container when a volume is created. tmpfs : Configures additional tmpfs options: size : The size for the tmpfs mount in bytes (either numeric or as bytes unit). mode : The file mode for the tmpfs mount as Unix permission bits as an octal number. consistency : The consistency requirements of the mount. Available values are platform specific. volumes_from volumes_from mounts all of the volumes from another service or container. You can optionally specify read-only access ro or read-write rw . If no access level is specified, then read-write access is used. You can also mount volumes from a container that is not managed by Compose by using the container: prefix. volumes_from: - service_name - service_name:ro - container:container_name - container:container_name:rw working_dir working_dir overrides the container's working directory which is specified by the image, for example Dockerfile's WORKDIR . Networks top-level element Networks are the layer that allow services to communicate with each other. The top-level networks element lets you configure named networks that can be reused across multiple services. To use a network across multiple services, you must explicitly grant each service access by using the networks attribute within the services top-level element. The networks top-level element has additional syntax that provides more granular control. Examples Basic example In the following example, at runtime, networks front-tier and back-tier are created and the frontend service is connected to front-tier and back-tier networks. services: frontend: image: example/webapp networks: - front-tier - back-tier networks: front-tier: back-tier: Advanced example services: proxy: build: ./proxy networks: - frontend app: build: ./app networks: - frontend - backend db: image: postgres networks: - backend networks: frontend: ## Use a custom driver driver: custom-driver-1 backend: ## Use a custom driver which takes special options driver: custom-driver-2 driver_opts: foo: \"1\" bar: \"2\" The advanced example shows a Compose file which defines two custom networks. The proxy service is isolated from the db service, because they do not share a network in common. Only app can talk to both. Attributes driver driver specifies which driver should be used for this network. Compose returns an error if the driver is not available on the platform. networks: db-data: driver: overlay Default and available values are platform specific. Compose supports the following drivers: none and host host : Use the host's networking stack. none : Turn off networking. host or none The syntax for using built-in networks such as host and none is different, as such networks implicitly exist outside the scope of Compose. To use them, you must define an external network with the name host or none and an alias that Compose can use ( hostnet and nonet in the following example), then grant the service access to that network using its alias. services: web: networks: hostnet: {} networks: hostnet: external: true name: host services: web: ... networks: nonet: {} networks: nonet: external: true name: none driver_opts driver_opts specifies a list of options as key-value pairs to pass to the driver. These options are driver-dependent. Consult the driver's documentation for more information. networks: db-data: driver_opts: foo: \"bar\" baz: 1 attachable If attachable is set to true , then standalone containers should be able to attach to this network, in addition to services. If a standalone container attaches to the network, it can communicate with services and other standalone containers that are also attached to the network. networks: mynet1: driver: overlay attachable: true enable_ipv6 enable_ipv6 enables IPv6 networking. For an example, see step four of Create an IPv6 network . external If set to true : - external specifies that this network\u2019s lifecycle is maintained outside of that of the application. Compose doesn't attempt to create these networks, and returns an error if one doesn't exist. - All other attributes apart from name are irrelevant. If Compose detects any other attribute, it rejects the Compose file as invalid. In the example below, proxy is the gateway to the outside world. Instead of attempting to create a network, Compose queries the platform for an existing network simply called outside and connects the proxy service's containers to it. services: proxy: image: example/proxy networks: - outside - default app: image: example/app networks: - default networks: outside: external: true ipam ipam specifies a custom IPAM configuration. This is an object with several properties, each of which is optional: driver : Custom IPAM driver, instead of the default. config : A list with zero or more configuration elements, each containing a: subnet : Subnet in CIDR format that represents a network segment ip_range : Range of IPs from which to allocate container IPs gateway : IPv4 or IPv6 gateway for the master subnet aux_addresses : Auxiliary IPv4 or IPv6 addresses used by Network driver, as a mapping from hostname to IP options : Driver-specific options as a key-value mapping. networks: mynet1: ipam: driver: default config: - subnet: 172.28.0.0/16 ip_range: 172.28.5.0/24 gateway: 172.28.5.254 aux_addresses: host1: 172.28.1.5 host2: 172.28.1.6 host3: 172.28.1.7 options: foo: bar baz: \"0\" internal By default, Compose provides external connectivity to networks. internal , when set to true , allows you to create an externally isolated network. labels Add metadata to containers using labels . You can use either an array or a dictionary. It is recommended that you use reverse-DNS notation to prevent labels from conflicting with those used by other software. networks: mynet1: labels: com.example.description: \"Financial transaction network\" com.example.department: \"Finance\" com.example.label-with-empty-value: \"\" networks: mynet1: labels: - \"com.example.description=Financial transaction network\" - \"com.example.department=Finance\" - \"com.example.label-with-empty-value\" Compose sets com.docker.compose.project and com.docker.compose.network labels. name name sets a custom name for the network. The name field can be used to reference networks which contain special characters. The name is used as is and is not scoped with the project name. networks: network1: name: my-app-net It can also be used in conjunction with the external property to define the platform network that Compose should retrieve, typically by using a parameter so the Compose file doesn't need to hard-code runtime specific values: networks: network1: external: true name: \"${NETWORK_ID}\" Volumes top-level element Volumes are persistent data stores implemented by the container engine. Compose offers a neutral way for services to mount volumes, and configuration parameters to allocate them to infrastructure. The top-level volumes declaration lets you configure named volumes that can be reused across multiple services. To use a volume across multiple services, you must explicitly grant each service access by using the volumes attribute within the services top-level element. The volumes attribute has additional syntax that provides more granular control. Example The following example shows a two-service setup where a database's data directory is shared with another service as a volume, named db-data , so that it can be periodically backed up. services: backend: image: example/database volumes: - db-data:/etc/data backup: image: backup-service volumes: - db-data:/var/lib/backup/data volumes: db-data: The db-data volume is mounted at the /var/lib/backup/data and /etc/data container paths for backup and backend respectively. Running docker compose up creates the volume if it doesn't already exist. Otherwise, the existing volume is used and is recreated if it's manually deleted outside of Compose. Attributes An entry under the top-level volumes section can be empty, in which case it uses the container engine's default configuration for creating a volume. Optionally, you can configure it with the following keys: driver Specifies which volume driver should be used. Default and available values are platform specific. If the driver is not available, Compose returns an error and doesn't deploy the application. volumes: db-data: driver: foobar driver_opts driver_opts specifies a list of options as key-value pairs to pass to the driver for this volume. The options are driver-dependent. volumes: example: driver_opts: type: \"nfs\" o: \"addr=10.40.0.199,nolock,soft,rw\" device: \":/docker/example\" external If set to true : - external specifies that this volume already exists on the platform and its lifecycle is managed outside of that of the application. Compose doesn't then create the volume, and returns an error if the volume doesn't exist. - All other attributes apart from name are irrelevant. If Compose detects any other attribute, it rejects the Compose file as invalid. In the example below, instead of attempting to create a volume called {project_name}_db-data , Compose looks for an existing volume simply called db-data and mounts it into the backend service's containers. services: backend: image: example/database volumes: - db-data:/etc/data volumes: db-data: external: true labels labels are used to add metadata to volumes. You can use either an array or a dictionary. It's recommended that you use reverse-DNS notation to prevent your labels from conflicting with those used by other software. volumes: db-data: labels: com.example.description: \"Database volume\" com.example.department: \"IT/Ops\" com.example.label-with-empty-value: \"\" volumes: db-data: labels: - \"com.example.description=Database volume\" - \"com.example.department=IT/Ops\" - \"com.example.label-with-empty-value\" Compose sets com.docker.compose.project and com.docker.compose.volume labels. name name sets a custom name for a volume. The name field can be used to reference volumes that contain special characters. The name is used as is and is not scoped with the stack name. volumes: db-data: name: \"my-app-data\" This makes it possible to make this lookup name a parameter of the Compose file, so that the model ID for the volume is hard-coded but the actual volume ID on the platform is set at runtime during deployment. For example, if DATABASE_VOLUME=my_volume_001 in your .env file: volumes: db-data: name: ${DATABASE_VOLUME} Running docker compose up uses the volume called my_volume_001 . It can also be used in conjunction with the external property. This means the name of the volume used to lookup the actual volume on the platform is set separately from the name used to refer to it within the Compose file: volumes: db-data: external: name: actual-name-of-volume Configs top-level element Configs allow services to adapt their behaviour without the need to rebuild a Docker image. Services can only access configs when explicitly granted by a configs attribute within the services top-level element. As with volumes, configs are mounted as files into a service's container's filesystem. The location of the mount point within the container defaults to / in Linux containers and C:\\ in Windows containers. By default, the config: - Is owned by the user running the container command but can be overridden by service configuration. - Has world-readable permissions (mode 0444), unless the service is configured to override this. The top-level configs declaration defines or references configuration data that is granted to services in your Compose application. The source of the config is either file or external . file : The config is created with the contents of the file at the specified path. environment : The config content is created with the value of an environment variable. content : The content is created with the inlined value. external : If set to true, external specifies that this config has already been created. Compose does not attempt to create it, and if it does not exist, an error occurs. name : The name of the config object in the container engine to look up. This field can be used to reference configs that contain special characters. The name is used as is and will not be scoped with the project name. Example 1 _http_config is created when the application is deployed, by registering the content of the httpd.conf as the configuration data. configs: http_config: file: ./httpd.conf Alternatively, http_config can be declared as external. Compose looks up http_config to expose the configuration data to relevant services. configs: http_config: external: true Example 2 _app_config is created when the application is deployed, by registering the inlined content as the configuration data. This comes with the benefits Compose will infer variables when creating the config, which allows to adjust content according to service configuration: configs: app_config: content: | debug=${DEBUG} spring.application.admin.enabled=${DEBUG} spring.application.name=${COMPOSE_PROJECT_NAME} Example 3 External configs lookup can also use a distinct key by specifying a name . The following example modifies the previous one to look up a config using the parameter HTTP_CONFIG_KEY . The the actual lookup key will is set at deployment time by the interpolation of variables, but exposed to containers as hard-coded ID http_config . configs: http_config: external: true name: \"${HTTP_CONFIG_KEY}\" If external is set to true , all other attributes apart from name are irrelevant. If Compose detecs any other attribute, it rejects the Compose file as invalid. Secrets top-level element Secrets are a flavor of Configs focusing on sensitive data, with specific constraint for this usage. Services can only access secrets when explicitly granted by a secrets attribute within the services top-level element. The top-level secrets declaration defines or references sensitive data that is granted to the services in your Compose application. The source of the secret is either file or environment . file : The secret is created with the contents of the file at the specified path. environment : The secret is created with the value of an environment variable. external : If set to true, external specifies that this secret has already been created. Compose does not attempt to create it, and if it does not exist, an error occurs. name : The name of the secret object in Docker. This field can be used to reference secrets that contain special characters. The name is used as is and isn't scoped with the project name. Example 1 server-certificate secret is created as _server-certificate when the application is deployed, by registering content of the server.cert as a platform secret. secrets: server-certificate: file: ./server.cert Example 2 token secret is created as _token when the application is deployed, by registering the content of the OAUTH_TOKEN environment variable as a platform secret. secrets: token: environment: \"OAUTH_TOKEN\" Alternatively, server-certificate can be declared as external. Compose looks up the server-certificate secret to expose to relevant services. secrets: server-certificate: external: true Example 3 External secrets lookup can also use a distinct key by specifying a name . The following example modifies the previous example to look up a secret using the name CERTIFICATE_KEY . The actual lookup key is set at deployment time by the interpolation of variables, but exposed to containers as hard-coded ID server-certificate . secrets: server-certificate: external: true name: \"${CERTIFICATE_KEY}\" If external is set to true , all other attributes apart from name are irrelevant. If Compose detects any other attribute, it rejects the Compose file as invalid. Your Compose file needs to explicitly grant access to the secrets to relevant services in your application. Fragments With Compose, you can use built-in YAML features to make your Compose file neater and more efficient. Anchors and aliases let you create re-usable blocks. This is useful if you start to find common configurations that span multiple services. Having re-usable blocks minimizes potential mistakes. Anchors are created using the & sign. The sign is followed by an alias name. You can use this alias with the * sign later to reference the value following the anchor. Make sure there is no space between the & and the * characters and the following alias name. You can use more than one anchor and alias in a single Compose file. Example 1 volumes: db-data: &default-volume driver: default metrics: *default-volume In the example above, a default-volume anchor is created based on the db-data volume. It is later reused by the alias *default-volume to define the metrics volume. Anchor resolution takes place before variables interpolation , so variables can't be used to set anchors or aliases. Example 2 services: first: image: my-image:latest environment: &env - CONFIG_KEY - EXAMPLE_KEY - DEMO_VAR second: image: another-image:latest environment: *env If you have an anchor that you want to use in more than one service, use it in conjunction with an extension to make your Compose file easier to maintain. Example 3 You may want to partially override values. Compose follows the rule outlined by YAML merge type . In the following example, metrics volume specification uses alias to avoid repetition but overrides name attribute: services: backend: image: example/database volumes: - db-data - metrics volumes: db-data: &default-volume driver: default name: \"data\" metrics: <<: *default-volume name: \"metrics\" Example 4 You can also extend the anchor to add additional values. services: first: image: my-image:latest environment: &env FOO: BAR ZOT: QUIX second: image: another-image:latest environment: <<: *env YET_ANOTHER: VARIABLE Note YAML merge only applies to mappings, and can't be used with sequences. In example above, the environment variables must be declared using the FOO: BAR mapping syntax, while the sequence syntax - FOO=BAR is only valid when no fragments are involved. Extension As with Fragments , Extensions can be used to make your Compose file more efficient and easier to maintain. Extensions can also be used with anchors and aliases . Use the prefix x- as a top-level element to modularize configurations that you want to reuse. Compose ignores any fields that start with x- , this is the sole exception where Compose silently ignores unrecognized fields. They also can be used within any structure in a Compose file where user-defined keys are not expected. Compose use those to enable experimental features, the same way browsers add support for custom CSS features Example 1 x-custom: foo: - bar - zot services: webapp: image: example/webapp x-foo: bar service: backend: deploy: placement: x-aws-role: \"arn:aws:iam::XXXXXXXXXXXX:role/foo\" x-aws-region: \"eu-west-3\" x-azure-region: \"france-central\" Example 2 x-env: &env environment: - CONFIG_KEY - EXAMPLE_KEY services: first: <<: *env image: my-image:latest second: <<: *env image: another-image:latest In this example, the environment variables do not belong to either of the services. They\u2019ve been lifted out completely into the x-env extension field. This defines a new node which contains the environment field. The &env YAML anchor is used so both services can reference the extension field\u2019s value as *env . Example 3 x-function: &function labels: function: \"true\" depends_on: - gateway networks: - functions deploy: placement: constraints: - 'node.platform.os == linux' services: ## Node.js gives OS info about the node (Host) nodeinfo: <<: *function image: functions/nodeinfo:latest environment: no_proxy: \"gateway\" https_proxy: $https_proxy ## Uses `cat` to echo back response, fastest function to execute. echoit: <<: *function image: functions/alpine:health environment: fprocess: \"cat\" no_proxy: \"gateway\" https_proxy: $https_proxy The nodeinfo and echoit services both include the x-function extension via the &function anchor, then set their specific image and environment. Example 4 Using YAML merge it is also possible to use multiple extensions and share and override additional attributes for specific needs: x-environment: &default-environment FOO: BAR ZOT: QUIX x-keys: &keys KEY: VALUE services: frontend: image: example/webapp environment: << : [*default-environment, *keys] YET_ANOTHER: VARIABLE Note YAML merge only applies to mappings, and can't be used with sequences. In the example above, the environment variables are declared using the FOO: BAR mapping syntax, while the sequence syntax - FOO=BAR is only valid when no fragments are involved. Informative Historical Notes This section is informative. At the time of writing, the following prefixes are known to exist: Prefix Vendor/Organization docker Docker kubernetes Kubernetes Specifying byte values Values express a byte value as a string in {amount}{byte unit} format: The supported units are b (bytes), k or kb (kilo bytes), m or mb (mega bytes) and g or gb (giga bytes). 2b 1024kb 2048k 300m 1gb Specifying durations Values express a duration as a string in the form of {value}{unit} . The supported units are us (microseconds), ms (milliseconds), s (seconds), m (minutes) and h (hours). Values can combine multiple values without separator. 10ms 40s 1m30s 1h5m30s20ms Interpolation Values in a Compose file can be set by variables and interpolated at runtime. Compose files use a Bash-like syntax ${VARIABLE} . Both $VARIABLE and ${VARIABLE} syntax is supported. Default values can be defined inline using typical shell syntax: ${VARIABLE:-default} evaluates to default if VARIABLE is unset or empty in the environment. ${VARIABLE-default} evaluates to default only if VARIABLE is unset in the environment. Similarly, the following syntax allows you to specify mandatory variables: ${VARIABLE:?err} exits with an error message containing err if VARIABLE is unset or empty in the environment. ${VARIABLE?err} exits with an error message containing err only if VARIABLE is unset in the environment. Interpolation can also be nested: ${VARIABLE:-${FOO}} ${VARIABLE?$FOO} ${VARIABLE:-${FOO:-default}} Other extended shell-style features, such as ${VARIABLE/foo/bar} , are not supported by Compose. You can use a $$ (double-dollar sign) when your configuration needs a literal dollar sign. This also prevents Compose from interpolating a value, so a $$ allows you to refer to environment variables that you don't want processed by Compose. web: build: . command: \"$$VAR_NOT_INTERPOLATED_BY_COMPOSE\" If Compose can't resolve a substituted variable and no default value is defined, it displays a warning and substitutes the variable with an empty string. As any values in a Compose file can be interpolated with variable substitution, including compact string notation for complex elements, interpolation is applied before a merge on a per-file basis. Interpolation applies only to YAML values, not to keys. For the few places where keys are actually arbitrary user-defined strings, such as labels or environment , an alternate equal sign syntax must be used for interpolation to apply. For example: services: foo: labels: \"$VAR_NOT_INTERPOLATED_BY_COMPOSE\": \"BAR\" services: foo: labels: - \"$VAR_INTERPOLATED_BY_COMPOSE=BAR\" Merge and override Compose lets you define a Compose application model through multiple Compose files . When doing so, Compose follows the rules declared in this section to merge Compose files. Mapping A YAML mapping gets merged by adding missing entries and merging the conflicting ones. Merging the following example YAML trees: services: foo: key1: value1 key2: value2 services: foo: key2: VALUE key3: value3 Results in a Compose application model equivalent to the YAML tree: services: foo: key1: value1 key2: VALUE key3: value3 Sequence A YAML sequence is merged by appending values from the overriding Compose file to the previous one. Merging the following example YAML trees: services: foo: DNS: - 1.1.1.1 services: foo: DNS: - 8.8.8.8 Results in a Compose application model equivalent to the YAML tree: services: foo: DNS: - 1.1.1.1 - 8.8.8.8 Exceptions Shell commands When merging Compose files that use the services attributes command , entrypoint and healthcheck: test , the value is overridden by the latest Compose file, and not appended. Merging the following example YAML trees: services: foo: command: [\"echo\", \"foo\"] services: foo: command: [\"echo\", \"bar\"] Results in a Compose application model equivalent to the YAML tree: services: foo: command: [\"echo\", \"bar\"] Unique resources Applies to the ports , volumes , secrets and configs services attributes. While these types are modeled in a Compose file as a sequence, they have special uniqueness requirements: Attribute Unique key volumes target secrets source configs source ports {ip, target, published, protocol} When merging Compose files, Compose appends new entries that do not violate a uniqueness constraint and merge entries that share a unique key. Merging the following example YAML trees: services: foo: volumes: - foo:/work services: foo: volumes: - bar:/work Results in a Compose application model equivalent to the YAML tree: services: foo: volumes: - bar:/work Reset value In addition to the previously described mechanism, an override Compose file can also be used to remove elements from your application model. For this purpose, the custom YAML tag !reset can be set to override a value set by the overriden Compose file. A valid value for attribute must be provided, but will be ignored and target attribute will be set with type's default value or null . For readability, it is recommended to explicitly set the attribute value to the null ( null ) or empty array [] (with !reset null or !reset [] ) so that it is clear that resulting attribute will be cleared. A base compose.yaml file: services: app: image: myapp ports: - \"8080:80\" environment: FOO: BAR And an overide.compose.yaml file: services: app: image: myapp ports: !reset [] environment: FOO: !reset null Results in: services: app: image: myapp Replace value While !reset can be used to remove a declaration from a Compose file using an override file, !override allows you to fully replace an attribute, bypassing the standard merge rules. A typical example is to fully replace a resource definition, to rely on a distinct model but using the same name. A base compose.yaml file: services: app: image: myapp ports: - \"8080:80\" To remove the original port, but expose a new one, the following override file is used: services: app: ports: !override - \"8443:443\" This results in: services: app: image: myapp ports: - \"8443:443\" If !override had not been used, both 8080:80 and 8443:443 would be exposed as per the merging rules outlined above . Include A Compose application can declare dependency on another Compose application. This is useful if: - You want to reuse other Compose files. - You need to factor out parts of your application model into separate Compose files so they can be managed separately or shared with others. - Teams need to keep a Compose file reasonably complicated for the limited amount of resources it has to declare for it's own sub-domain, within a larger deployment. The include top-level section is used to define the dependency on another Compose application, or sub-domain. Each path listed in the include section is loaded as an individual Compose application model, with it's own project directory, in order to resolve relative paths. Once the included Compose application is loaded, all resources definitions are copied into the current Compose application model. Compose displays a warning if resource names conflict and doesn't try to merge them. To enforce this, include is evaluated after the Compose file(s) selected to define the Compose application model have been parsed and merged, so that conflicts between Compose files are detected. include applies recursively so an included Compose file which declares its own include section, triggers those other files to be included as well. Any volumes, networks, or other resources pulled in from the included Compose file can be used by the current Compose application for cross-service references. For example: include: - my-compose-include.yaml #with serviceB declared services: serviceA: build: . depends_on: - serviceB #use serviceB directly as if it was declared in this Compose file Compose also supports the use of interpolated variables with include . It's recommended that you specify mandatory variables . For example: include: -${INCLUDE_PATH:?FOO}/compose.yaml Short syntax The short syntax only defines paths to other Compose files. The file is loaded with the parent folder as the project directory, and an optional .env file that is loaded to define any variables' default values by interpolation. The local project's environment can override those values. include: - ../commons/compose.yaml - ../another_domain/compose.yaml services: webapp: depends_on: - included-service ## defined by another_domain In the above example, both ../commons/compose.yaml and ../another_domain/compose.yaml are loaded as individual Compose projects. Relative paths in Compose files being referred by include are resolved relative to their own Compose file path, not based on the local project's directory. Variables are interpolated using values set in the optional .env file in same folder, and is overridden by the local project's environment. Long syntax The long syntax offers more control over the sub-project parsing: include: - path: ../commons/compose.yaml project_directory: .. env_file: ../another/.env path path is required and defines the location of the Compose file(s) to be parsed and included into the local Compose model. path can be set either to a string when a single Compose file is involved, or to a list of strings when multiple Compose files need to be merged together to define the Compose model to be included in the local application. include: - path: - ../commons/compose.yaml - ./commons-override.yaml project_directory project_directory defines a base path to resolve relative paths set in the Compose file. It defaults to the directory of the included Compose file. env_file env_file defines an environment file(s) to use to define default values when interpolating variables in the Compose file being parsed. It defaults to .env file in the project_directory for the Compose file being parsed. env_file can be set either to a string or a list of strings when multiple environment files need to be merged to define a project environment. include: - path: ../another/compose.yaml env_file: - ../another/.env - ../another/dev.env The local project's environment has precedence over the values set by the Compose file, so that the local project can override values for customization. Profiles With profiles you can define a set of active profiles so your Compose application model is adjusted for various usages and environments. The exact mechanism is implementation specific and may include command line flags, environment variables, etc. The services top-level element supports a profiles attribute to define a list of named profiles. Services without a profiles attribute are always enabled. A service is ignored by Compose when none of the listed profiles match the active ones, unless the service is explicitly targeted by a command. In that case its profile is added to the set of active profiles. Note All other top-level elements are not affected by profiles and are always active. References to other services (by links , extends or shared resource syntax service:xxx ) do not automatically enable a component that would otherwise have been ignored by active profiles. Instead Compose returns an error. Illustrative example services: foo: image: foo bar: image: bar profiles: - test baz: image: baz depends_on: - bar profiles: - test zot: image: zot depends_on: - bar profiles: - debug In the above example: If the Compose application model is parsed with no profile enabled, it only contains the foo service. If the profile test is enabled, the model contains the services bar and baz , and service foo , which is always enabled. If the profile debug is enabled, the model contains both foo and zot services, but not bar and baz , and as such the model is invalid regarding the depends_on constraint of zot . If the profiles debug and test are enabled, the model contains all services; foo , bar , baz and zot . If Compose is executed with bar as the explicit service to run, bar and the test profile are active even if test profile is not enabled. If Compose is executed with baz as the explicit service to run, the service baz and the profile test are active and bar is pulled in by the depends_on constraint. If Compose is executed with zot as the explicit service to run, again the model is invalid regarding the depends_on constraint of zot , since zot and bar have no common profiles listed. If Compose is executed with zot as the explicit service to run and profile test is enabled, profile debug is automatically enabled and service bar is pulled in as a dependency starting both services zot and bar . See how you can use profiles in Docker Compose .","title":"Docker"},{"location":"docker/#docker","text":"sudo apt update && sudo apt upgrade -y sudo apt install git curl htop mc rsync zsh-autosuggestions sudo apt install zsh chsh -s $(which zsh) sh -c \"$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)\" rsync -avuPz ./.oh-my-zsh/custom/themes/custom.zsh-theme furyhawk@arm:/home/furyhawk/.oh-my-zsh/custom/themes/custom.zsh-theme git clone https://github.com/zsh-users/zsh-autosuggestions.git $ZSH_CUSTOM/plugins/zsh-autosuggestions git clone https://github.com/zdharma-continuum/fast-syntax-highlighting.git ${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/plugins/fast-syntax-highlighting ## Enable plugins by adding them to .zshrc. - Open .zshrc `nano ~/.zshrc` - ZSH_THEME=\"custom\" - Find the line which says `plugins=(git)`. - Replace that line with `plugins=(git zsh-autosuggestions zsh-syntax-highlighting fast-syntax-highlighting zsh-autocomplete)` touch .zprofile nano .zprofile # set PATH so it includes user's private bin if it exists if [ -d \"$HOME/bin\" ] ; then PATH=\"$HOME/bin:$PATH\" fi # set PATH so it includes user's private bin if it exists if [ -d \"$HOME/.local/bin\" ] ; then PATH=\"$HOME/.local/bin:$PATH\" fi export TZ=Asia/Singapore export NODE_ID=$(docker info -f '{{.Swarm.NodeID}}') export EMAIL=furyx@hotmail.com export DOMAIN=furyhawk.lol sudo nano /etc/fstab touch .credentials nano .credentials sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc sudo chmod a+r /etc/apt/keyrings/docker.asc echo \\ \"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \\ $(. /etc/os-release && echo \"$VERSION_CODENAME\") stable\" | \\ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin # sudo groupadd docker sudo usermod -aG docker $USER sudo systemctl enable docker.service sudo systemctl enable containerd.service systemctl list-units --type=service --state=active cd /etc/docker sudo touch daemon.json sudo nano daemon.json { \"log-driver\": \"json-file\", \"log-opts\": { \"max-size\": \"10m\", \"max-file\": \"3\" } } docker swarm join --token SWMTKN-1-xxx 192.168.50.114:2377 docker node update --availability drain node docker node update --availability Active node curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm chmod +x cephadm sudo ./cephadm add-repo --release octopus sudo ./cephadm install cephadm install ceph-common mkdir -p /etc/ceph sudo ./cephadm bootstrap --mon-ip 192.168.65.19 URL: https://debian.local:8443/ User: admin Password: 12345678 You can access the Ceph CLI with: sudo ./cephadm shell --fsid 8ad7deb6-265a-11ef-9e81-c2d0c41fc7e0 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring Please consider enabling telemetry to help improve Ceph: ceph telemetry on For more information see: https://docs.ceph.com/docs/master/mgr/telemetry/ alias ceph='./cephadm shell -- ceph' sudo ceph -v ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) sudo ceph orch host label add debian mon Added label mon to host debian sudo ceph orch apply mon debian Scheduled mon update... sudo ceph orch host ls ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-2 ssh-copy-id -f -i /etc/ceph/ceph.pub osd1 ssh-copy-id -f -i /etc/ceph/ceph.pub osd2 ssh-copy-id -f -i /etc/ceph/ceph.pub osd3 sudo ceph orch host add osd1 sudo ceph orch host add osd2 sudo ceph orch host add osd3 sudo ceph orch apply osd --all-available-devices sudo ceph status sudo find / -iname 'cephadm*' 2>/dev/null systemctl --user start docker-desktop sudo groupadd docker sudo usermod -aG docker $USER groups ${USER} curl -fsSL test.docker.com -o get-docker.sh && sh get-docker.sh sudo apt-get install libffi-dev libssl-dev sudo apt install python3-dev sudo apt-get install -y python3 python3-pip pip install \"cython<3.0.0\" wheel pip install \"pyyaml==5.4.1\" --no-build-isolation pip install docker-compose sudo systemctl enable docker sudo apt-get purge docker-ce sudo apt-get purge docker-ce-cli sudo rm -rf /var/lib/docker rootless sudo apt install -y dbus-user-session sudo apt install -y fuse-overlayfs sudo apt install -y fuse-overlayfs sudo systemctl disable --now docker.service docker.socket sudo rm -rf /var/lib/docker sudo apt-get purge docker-ce docker-ce-cli containerd.io sudo apt-get purge docker-ce docker-ce-cli containerd.io apt-get install -y uidmap curl -fsSL https://get.docker.com/rootless | sh nano .zshrc export PATH=/home/furyhawk/bin:$PATH export DOCKER_HOST=unix:///run/user/1000/docker.sock systemctl --user enable docker sudo loginctl enable-linger $(whoami) docker context use rootless sudo setcap cap_net_bind_service=ep $(which rootlesskit) systemctl --user restart docker #sudo apt-get install -y docker-ce-rootless-extras dockerd-rootless-setuptool.sh uninstall cd ~/bin rm -f containerd containerd-shim containerd-shim-runc-v2 ctr docker docker-init docker-proxy dockerd dockerd-rootless-setuptool.sh dockerd-rootless.sh rootlesskit rootlesskit-docker-proxy runc vpnkit netstat -ltup","title":"Docker"},{"location":"docker/#the-compose-specification","text":"{:.no_toc} ToC {:toc}","title":"The Compose Specification"},{"location":"docker/#status-of-this-document","text":"This document specifies the Compose file format used to define multi-containers applications. Distribution of this document is unlimited.","title":"Status of this document"},{"location":"docker/#requirements-and-optional-attributes","text":"The Compose specification includes properties designed to target a local OCI container runtime, exposing Linux kernel specific configuration options, but also some Windows container specific properties. It is also designed for cloud platform features related to resource placement on a cluster, replicated application distribution, and scalability. We acknowledge that no Compose implementation is expected to support all attributes, and that support for some properties is platform dependent and can only be confirmed at runtime. The definition of a versioned schema to control the supported properties in a Compose file, established by the docker-compose tool where the Compose file format was designed, doesn't offer any guarantee to the end-user that attributes will be actually implemented. The specification defines the expected configuration syntax and behavior. Unless noted, supporting any of these is optional. A Compose implementation to parse a Compose file using unsupported attributes should warn users. We recommend the following implementors to support those running modes: Default: warn the user about unsupported attributes, but ignore them Strict: warn the user about unsupported attributes and reject the Compose file Loose: ignore unsupported attributes AND unknown attributes (that were not defined by the spec by the time implementation was created) From this point onwards, references made to 'Compose' can be interpreted as 'a Compose implementation'.","title":"Requirements and optional attributes"},{"location":"docker/#the-compose-application-model","text":"The Compose Specification lets you define a platform-agnostic container based application. Such an application is designed as a set of containers which have to both run together with adequate shared resources and communication channels. Computing components of an application are defined as services . A service is an abstract concept implemented on platforms by running the same container image, and configuration, one or more times. Services communicate with each other through networks . In the Compose Specification, a network is a platform capability abstraction to establish an IP route between containers within services connected together. Low-level, platform-specific networking options are grouped into the Network definition and may be partially implemented on some platforms. Services store and share persistent data into volumes . The Specification describes such a persistent data as a high-level filesystem mount with global options. Actual platform-specific implementation details are grouped into the volumes definition and may be partially implemented on some platforms. Some services require configuration data that is dependent on the runtime or platform. For this, the Specification defines a dedicated configs concept. From a service container point of view, configs are comparable to volumes, in that they are files mounted into the container. But the actual definition involves distinct platform resources and services, which are abstracted by this type. A secret is a specific flavor of configuration data for sensitive data that should not be exposed without security considerations. Secrets are made available to services as files mounted into their containers, but the platform-specific resources to provide sensitive data are specific enough to deserve a distinct concept and definition within the Compose specification. Note With volumes, configs and secrets you can have a simple declaration at the top-level and then add more platform-specific information at the service level. A project is an individual deployment of an application specification on a platform. A project's name, set with the top-level name attribute, is used to group resources together and isolate them from other applications or other installation of the same Compose specified application with distinct parameters. If you are creating resources on a platform, you must prefix resource names by project and set the label com.docker.compose.project . Compose offers a way for users to set a custom project name and override this name, so that the same compose.yaml file can be deployed twice on the same infrastructure, without changes, by just passing a distinct name. Project names must contain only lowercase letters, decimal digits, dashes, and underscores, and must begin with a lowercase letter or decimal digit.","title":"The Compose application model"},{"location":"docker/#illustrative-example","text":"The following example illustrates the Compose Specification concepts outlined above. The example is non-normative. Consider an application split into a frontend web application and a backend service. The frontend is configured at runtime with an HTTP configuration file managed by infrastructure, providing an external domain name, and an HTTPS server certificate injected by the platform's secured secret store. The backend stores data in a persistent volume. Both services communicate with each other on an isolated back-tier network, while the frontend is also connected to a front-tier network and exposes port 443 for external usage. %%{ init: { 'flowchart': { 'curve': 'linear' } } }%% flowchart LR subgraph A[INFRASTRUCTURE] direction TB subgraph TOP[\" \"] subgraph B1[Frontend Service] fs[\"`**webapp**`\"] end style B1 fill:#ccd6e8, stroke-width:0px subgraph B2[Backend Service] bs[\"`**database**`\"] end style B2 fill:#ccd6e8, stroke-width:0px end style TOP fill:transparent, stroke-width:2px, stroke:#62affb, stroke-dasharray: 5 5 key[ro= read only\\nr+w = read write] style key fill:transparent, stroke-width:0px,text-align: left, size: 94px direction TB id2(Server\\nCertificate) id1(HTTP\\nConfiguration) id1 & id2 -.-|ro| B1 style id1 stroke:#000,stroke-width:1px,stroke-dasharray: 10 style id2 stroke:#000,stroke-width:1px,stroke-dasharray: 10 B2 ==r+w==> id3[(Persistent\\nVolume)] end style A fill:#eeeeee, stroke-width:0px direction LR id4[External\\nUser] ---id5(((443)))--->|Frontend\\nNetwork| B1 style id4 stroke:#000,stroke-width:2px B1 --Backend\\nNetwork--> B2 The example application is composed of the following parts: 2 services, backed by Docker images: webapp and database 1 secret (HTTPS certificate), injected into the frontend 1 configuration (HTTP), injected into the frontend 1 persistent volume, attached to the backend 2 networks services: frontend: image: example/webapp ports: - \"443:8043\" networks: - front-tier - back-tier configs: - httpd-config secrets: - server-certificate backend: image: example/database volumes: - db-data:/etc/data networks: - back-tier volumes: db-data: driver: flocker driver_opts: size: \"10GiB\" configs: httpd-config: external: true secrets: server-certificate: external: true networks: ## The presence of these objects is sufficient to define them front-tier: {} back-tier: {} This example illustrates the distinction between volumes, configs and secrets. While all of them are all exposed to service containers as mounted files or directories, only a volume can be configured for read+write access. Secrets and configs are read-only. The volume configuration allows you to select a volume driver and pass driver options to tweak volume management according to the actual infrastructure. Configs and secrets rely on platform services, and are declared external as they are not managed as part of the application lifecycle. Compose uses a platform-specific lookup mechanism to retrieve runtime values.","title":"Illustrative example"},{"location":"docker/#compose-file","text":"The Compose file is a YAML file defining: - Version (Optional) - Services (Required) - Networks - Volumes - Configs - Secrets The default path for a Compose file is compose.yaml (preferred) or compose.yml that is placed in the working directory. Compose also supports docker-compose.yaml and docker-compose.yml for backwards compatibility of earlier versions. If both files exist, Compose prefers the canonical compose.yaml . You can use fragments and extensions to keep your Compose file efficient and easy to maintain. Multiple Compose files can be merged together to define the application model. The combination of YAML files are implemented by appending or overriding YAML elements based on the Compose file order you set. Simple attributes and maps get overridden by the highest order Compose file, lists get merged by appending. Relative paths are resolved based on the first Compose file's parent folder, whenever complimentary files being merged are hosted in other folders. As some Compose file elements can both be expressed as single strings or complex objects, merges apply to the expanded form. If you want to reuse other Compose files, or factor out parts of you application model into separate Compose files, you can also use include . This is useful if your Compose application is dependent on another application which is managed by a different team, or needs to be shared with others.","title":"Compose file"},{"location":"docker/#version-and-name-top-level-elements","text":"","title":"Version and name top-level elements"},{"location":"docker/#version-top-level-element","text":"The top-level version property is defined by the Compose Specification for backward compatibility. It is only informative. Compose doesn't use version to select an exact schema to validate the Compose file, but prefers the most recent schema when it's implemented. Compose validates whether it can fully parse the Compose file. If some fields are unknown, typically because the Compose file was written with fields defined by a newer version of the Specification, you'll receive a warning message. Compose offers options to ignore unknown fields (as defined by \"loose\" mode).","title":"Version top-level element"},{"location":"docker/#name-top-level-element","text":"The top-level name property is defined by the Specification as the project name to be used if you don't set one explicitly. Compose offers a way for you to override this name, and sets a default project name to be used if the top-level name element is not set. Whenever a project name is defined by top-level name or by some custom mechanism, it is exposed for interpolation and environment variable resolution as COMPOSE_PROJECT_NAME services: foo: image: busybox environment: - COMPOSE_PROJECT_NAME command: echo \"I'm running ${COMPOSE_PROJECT_NAME}\"","title":"Name top-level element"},{"location":"docker/#services-top-level-element","text":"A service is an abstract definition of a computing resource within an application which can be scaled or replaced independently from other components. Services are backed by a set of containers, run by the platform according to replication requirements and placement constraints. As services are backed by containers, they are defined by a Docker image and set of runtime arguments. All containers within a service are identically created with these arguments. A Compose file must declare a services top-level element as a map whose keys are string representations of service names, and whose values are service definitions. A service definition contains the configuration that is applied to each service container. Each service may also include a build section, which defines how to create the Docker image for the service. Compose supports building docker images using this service definition. If not used, the build section is ignored and the Compose file is still considered valid. Build support is an optional aspect of the Compose Specification, and is described in detail in the Compose Build Specification documentation. Each service defines runtime constraints and requirements to run its containers. The deploy section groups these constraints and allows the platform to adjust the deployment strategy to best match containers' needs with available resources. Deploy support is an optional aspect of the Compose Specification, and is described in detail in the Compose Deploy Specification documentation. If not implemented the deploy section is ignored and the Compose file is still considered valid.","title":"Services top-level element"},{"location":"docker/#attach","text":"When attach is defined and set to false Compose does not collect service logs, until you explicitly request it to. The default service configuration is attach: true .","title":"attach"},{"location":"docker/#build","text":"build specifies the build configuration for creating a container image from source, as defined in the Compose Build Specification .","title":"build"},{"location":"docker/#blkio_config","text":"blkio_config defines a set of configuration options to set block IO limits for a service. services: foo: image: busybox blkio_config: weight: 300 weight_device: - path: /dev/sda weight: 400 device_read_bps: - path: /dev/sdb rate: '12mb' device_read_iops: - path: /dev/sdb rate: 120 device_write_bps: - path: /dev/sdb rate: '1024k' device_write_iops: - path: /dev/sdb rate: 30","title":"blkio_config"},{"location":"docker/#device_read_bps-device_write_bps","text":"Set a limit in bytes per second for read / write operations on a given device. Each item in the list must have two keys: path : Defines the symbolic path to the affected device. rate : Either as an integer value representing the number of bytes or as a string expressing a byte value.","title":"device_read_bps, device_write_bps"},{"location":"docker/#device_read_iops-device_write_iops","text":"Set a limit in operations per second for read / write operations on a given device. Each item in the list must have two keys: path : Defines the symbolic path to the affected device. rate : As an integer value representing the permitted number of operations per second.","title":"device_read_iops, device_write_iops"},{"location":"docker/#weight","text":"Modify the proportion of bandwidth allocated to a service relative to other services. Takes an integer value between 10 and 1000, with 500 being the default.","title":"weight"},{"location":"docker/#weight_device","text":"Fine-tune bandwidth allocation by device. Each item in the list must have two keys: path : Defines the symbolic path to the affected device. weight : An integer value between 10 and 1000.","title":"weight_device"},{"location":"docker/#cpu_count","text":"cpu_count defines the number of usable CPUs for service container.","title":"cpu_count"},{"location":"docker/#cpu_percent","text":"cpu_percent defines the usable percentage of the available CPUs.","title":"cpu_percent"},{"location":"docker/#cpu_shares","text":"cpu_shares defines, as integer value, a service container's relative CPU weight versus other containers.","title":"cpu_shares"},{"location":"docker/#cpu_period","text":"cpu_period configures CPU CFS (Completely Fair Scheduler) period when a platform is based on Linux kernel.","title":"cpu_period"},{"location":"docker/#cpu_quota","text":"cpu_quota configures CPU CFS (Completely Fair Scheduler) quota when a platform is based on Linux kernel.","title":"cpu_quota"},{"location":"docker/#cpu_rt_runtime","text":"cpu_rt_runtime configures CPU allocation parameters for platforms with support for realtime scheduler. It can be either an integer value using microseconds as unit or a duration . cpu_rt_runtime: '400ms' cpu_rt_runtime: 95000`","title":"cpu_rt_runtime"},{"location":"docker/#cpu_rt_period","text":"cpu_rt_period configures CPU allocation parameters for platforms with support for realtime scheduler. It can be either an integer value using microseconds as unit or a duration . cpu_rt_period: '1400us' cpu_rt_period: 11000`","title":"cpu_rt_period"},{"location":"docker/#cpus","text":"DEPRECATED: use deploy.limits.cpus cpus define the number of (potentially virtual) CPUs to allocate to service containers. This is a fractional number. 0.000 means no limit.","title":"cpus"},{"location":"docker/#cpuset","text":"cpuset defines the explicit CPUs in which to allow execution. Can be a range 0-3 or a list 0,1","title":"cpuset"},{"location":"docker/#cap_add","text":"cap_add specifies additional container capabilities as strings. cap_add: - ALL","title":"cap_add"},{"location":"docker/#cap_drop","text":"cap_drop specifies container capabilities to drop as strings. cap_drop: - NET_ADMIN - SYS_ADMIN","title":"cap_drop"},{"location":"docker/#cgroup","text":"cgroup specifies the cgroup namespace to join. When unset, it is the container runtime's decision to select which cgroup namespace to use, if supported. host : Runs the container in the Container runtime cgroup namespace. private : Runs the container in its own private cgroup namespace.","title":"cgroup"},{"location":"docker/#cgroup_parent","text":"cgroup_parent specifies an optional parent cgroup for the container. cgroup_parent: m-executor-abcd","title":"cgroup_parent"},{"location":"docker/#command","text":"command overrides the default command declared by the container image, for example by Dockerfile's CMD . command: bundle exec thin -p 3000 The value can also be a list, in a manner similar to Dockerfile : command: [ \"bundle\", \"exec\", \"thin\", \"-p\", \"3000\" ] If the value is null , the default command from the image is used. If the value is [] (empty list) or '' (empty string), the default command declared by the image is ignored, i.e. overridden to be empty.","title":"command"},{"location":"docker/#configs","text":"Configs allow services to adapt their behaviour without the need to rebuild a Docker image. Services can only access configs when explicitly granted by the configs attribute. Two different syntax variants are supported. Compose reports an error if config doesn't exist on the platform or isn't defined in the configs top-level element in the Compose file. There are two syntaxes defined for configs. To remain compliant to this specification, an implementation must support both syntaxes. Implementations must allow use of both short and long syntaxes within the same document. You can grant a service access to multiple configs, and you can mix long and short syntax.","title":"configs"},{"location":"docker/#short-syntax","text":"The short syntax variant only specifies the config name. This grants the container access to the config and mounts it as files into a service\u2019s container\u2019s filesystem. The location of the mount point within the container defaults to / in Linux containers, and C:\\ in Windows containers. The following example uses the short syntax to grant the redis service access to the my_config and my_other_config configs. The value of my_config is set to the contents of the file ./my_config.txt , and my_other_config is defined as an external resource, which means that it has already been defined in the platform. If the external config does not exist, the deployment fails. services: redis: image: redis:latest configs: - my_config - my_other_config configs: my_config: file: ./my_config.txt my_other_config: external: true","title":"Short syntax"},{"location":"docker/#long-syntax","text":"The long syntax provides more granularity in how the config is created within the service's task containers. source : The name of the config as it exists in the platform. target : The path and name of the file to be mounted in the service's task containers. Defaults to / if not specified. uid and gid : The numeric UID or GID that owns the mounted config file within the service's task containers. Default value when not specified is USER running container. mode : The permissions for the file that is mounted within the service's task containers, in octal notation. Default value is world-readable ( 0444 ). Writable bit must be ignored. The executable bit can be set. The following example sets the name of my_config to redis_config within the container, sets the mode to 0440 (group-readable) and sets the user and group to 103 . The redis service does not have access to the my_other_config config. services: redis: image: redis:latest configs: - source: my_config target: /redis_config uid: \"103\" gid: \"103\" mode: 0440 configs: my_config: external: true my_other_config: external: true","title":"Long syntax"},{"location":"docker/#container_name","text":"container_name is a string that specifies a custom container name, rather than a name generated by default. container_name: my-web-container Compose does not scale a service beyond one container if the Compose file specifies a container_name . Attempting to do so results in an error. container_name follows the regex format of [a-zA-Z0-9][a-zA-Z0-9_.-]+","title":"container_name"},{"location":"docker/#credential_spec","text":"credential_spec configures the credential spec for a managed service account. If you have services that use Windows containers, you can use file: and registry: protocols for credential_spec . Compose also supports additional protocols for custom use-cases. The credential_spec must be in the format file:// or registry:// . credential_spec: file: my-credential-spec.json When using registry: , the credential spec is read from the Windows registry on the daemon's host. A registry value with the given name must be located in: HKLM\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Virtualization\\Containers\\CredentialSpecs The following example loads the credential spec from a value named my-credential-spec in the registry: credential_spec: registry: my-credential-spec","title":"credential_spec"},{"location":"docker/#example-gmsa-configuration","text":"When configuring a gMSA credential spec for a service, you only need to specify a credential spec with config , as shown in the following example: services: myservice: image: myimage:latest credential_spec: config: my_credential_spec configs: my_credentials_spec: file: ./my-credential-spec.json|","title":"Example gMSA configuration"},{"location":"docker/#depends_on","text":"depends_on expresses startup and shutdown dependencies between services.","title":"depends_on"},{"location":"docker/#short-syntax_1","text":"The short syntax variant only specifies service names of the dependencies. Service dependencies cause the following behaviors: Compose creates services in dependency order. In the following example, db and redis are created before web . Compose removes services in dependency order. In the following example, web is removed before db and redis . Simple example: services: web: build: . depends_on: - db - redis redis: image: redis db: image: postgres Compose guarantees dependency services have been started before starting a dependent service. Compose waits for dependency services to be \"ready\" before starting a dependent service.","title":"Short syntax"},{"location":"docker/#long-syntax_1","text":"The long form syntax enables the configuration of additional fields that can't be expressed in the short form. restart : When set to true Compose restarts this service after it updates the dependency service. This applies to an explicit restart controlled by a Compose operation, and excludes automated restart by the container runtime after the container dies. condition : Sets the condition under which dependency is considered satisfied service_started : An equivalent of the short syntax described above service_healthy : Specifies that a dependency is expected to be \"healthy\" (as indicated by healthcheck ) before starting a dependent service. service_completed_successfully : Specifies that a dependency is expected to run to successful completion before starting a dependent service. required : When set to false Compose only warns you when the dependency service isn't started or available. If it's not defined the default value of required is true . Service dependencies cause the following behaviors: Compose creates services in dependency order. In the following example, db and redis are created before web . Compose waits for healthchecks to pass on dependencies marked with service_healthy . In the following example, db is expected to be \"healthy\" before web is created. Compose removes services in dependency order. In the following example, web is removed before db and redis . services: web: build: . depends_on: db: condition: service_healthy restart: true redis: condition: service_started redis: image: redis db: image: postgres Compose guarantees dependency services are started before starting a dependent service. Compose guarantees dependency services marked with service_healthy are \"healthy\" before starting a dependent service.","title":"Long syntax"},{"location":"docker/#deploy","text":"deploy specifies the configuration for the deployment and lifecycle of services, as defined in the Compose Deploy Specification .","title":"deploy"},{"location":"docker/#develop","text":"develop specifies the development configuration for maintaining a container in sync with source, as defined in the Development Section .","title":"develop"},{"location":"docker/#device_cgroup_rules","text":"device_cgroup_rules defines a list of device cgroup rules for this container. The format is the same format the Linux kernel specifies in the Control Groups Device Whitelist Controller . device_cgroup_rules: - 'c 1:3 mr' - 'a 7:* rmw'","title":"device_cgroup_rules"},{"location":"docker/#devices","text":"devices defines a list of device mappings for created containers in the form of HOST_PATH:CONTAINER_PATH[:CGROUP_PERMISSIONS] . devices: - \"/dev/ttyUSB0:/dev/ttyUSB0\" - \"/dev/sda:/dev/xvda:rwm\"","title":"devices"},{"location":"docker/#dns","text":"dns defines custom DNS servers to set on the container network interface configuration. It can be a single value or a list. dns: 8.8.8.8 dns: - 8.8.8.8 - 9.9.9.9","title":"dns"},{"location":"docker/#dns_opt","text":"dns_opt list custom DNS options to be passed to the container\u2019s DNS resolver ( /etc/resolv.conf file on Linux). dns_opt: - use-vc - no-tld-query","title":"dns_opt"},{"location":"docker/#dns_search","text":"dns_search defines custom DNS search domains to set on container network interface configuration. It can be a single value or a list. dns_search: example.com dns_search: - dc1.example.com - dc2.example.com","title":"dns_search"},{"location":"docker/#domainname","text":"domainname declares a custom domain name to use for the service container. It must be a valid RFC 1123 hostname.","title":"domainname"},{"location":"docker/#entrypoint","text":"entrypoint declares the default entrypoint for the service container. This overrides the ENTRYPOINT instruction from the service's Dockerfile. If entrypoint is non-null, Compose ignores any default command from the image, for example the CMD instruction in the Dockerfile. See also command to set or override the default command to be executed by the entrypoint process. In its short form, the value can be defined as a string: entrypoint: /code/entrypoint.sh Alternatively, the value can also be a list, in a manner similar to the Dockerfile : entrypoint: - php - -d - zend_extension=/usr/local/lib/php/extensions/no-debug-non-zts-20100525/xdebug.so - -d - memory_limit=-1 - vendor/bin/phpunit If the value is null , the default entrypoint from the image is used. If the value is [] (empty list) or '' (empty string), the default entrypoint declared by the image is ignored, i.e. overridden to be empty.","title":"entrypoint"},{"location":"docker/#env_file","text":"env_file adds environment variables to the container based on the file content. env_file: .env env_file can also be a list. The files in the list are processed from the top down. For the same variable specified in two env files, the value from the last file in the list stands. env_file: - ./a.env - ./b.env List elements can also be declared as a mapping, which then lets you set an additional attribute required . This defaults to true . When required is set to false and the .env file is missing, Compose silently ignores the entry. env_file: - path: ./default.env required: true ## default - path: ./override.env required: false Relative path are resolved from the Compose file's parent folder. As absolute paths prevent the Compose file from being portable, Compose warns you when such a path is used to set env_file . Environment variables declared in the environment section override these values. This holds true even if those values are empty or undefined.","title":"env_file"},{"location":"docker/#env_file-format","text":"Each line in an .env file must be in VAR[=[VAL]] format. The following syntax rules apply: Lines beginning with # are processed as comments and ignored. Blank lines are ignored. Unquoted and double-quoted ( \" ) values have Interpolation applied. Each line represents a key-value pair. Values can optionally be quoted. VAR=VAL -> VAL VAR=\"VAL\" -> VAL VAR='VAL' -> VAL Inline comments for unquoted values must be preceded with a space. VAR=VAL ## comment -> VAL VAR=VAL## not a comment -> VAL## not a comment Inline comments for quoted values must follow the closing quote. VAR=\"VAL ## not a comment\" -> VAL ## not a comment VAR=\"VAL\" ## comment -> VAL Single-quoted ( ' ) values are used literally. VAR='$OTHER' -> $OTHER VAR='${OTHER}' -> ${OTHER} Quotes can be escaped with \\ . VAR='Let\\'s go!' -> Let's go! VAR=\"{\\\"hello\\\": \\\"json\\\"}\" -> {\"hello\": \"json\"} Common shell escape sequences including \\n , \\r , \\t , and \\\\ are supported in double-quoted values. VAR=\"some\\tvalue\" -> some value VAR='some\\tvalue' -> some\\tvalue VAR=some\\tvalue -> some\\tvalue VAL may be omitted, in such cases the variable value is an empty string. =VAL may be omitted, in such cases the variable is unset. ## Set Rails/Rack environment RACK_ENV=development VAR=\"quoted\"","title":"Env_file format"},{"location":"docker/#environment","text":"environment defines environment variables set in the container. environment can use either an array or a map. Any boolean values; true, false, yes, no, should be enclosed in quotes to ensure they are not converted to True or False by the YAML parser. Environment variables can be declared by a single key (no value to equals sign). In this case Compose relies on you to resolve the value. If the value is not resolved, the variable is unset and is removed from the service container environment. Map syntax: environment: RACK_ENV: development SHOW: \"true\" USER_INPUT: Array syntax: environment: - RACK_ENV=development - SHOW=true - USER_INPUT When both env_file and environment are set for a service, values set by environment have precedence.","title":"environment"},{"location":"docker/#expose","text":"expose defines the (incoming) port or a range of ports that Compose exposes from the container. These ports must be accessible to linked services and should not be published to the host machine. Only the internal container ports can be specified. Syntax is /[] or /[] for a port range. When not explicitly set, tcp protocol is used. expose: - \"3000\" - \"8000\" - \"8080-8085/tcp Note If the Dockerfile for the image already exposes ports, it is visible to other containers on the network even if expose is not set in your Compose file.","title":"expose"},{"location":"docker/#extends","text":"extends lets you share common configurations among different files, or even different projects entirely. With extends you can define a common set of service options in one place and refer to it from anywhere. You can refer to another Compose file and select a service you want to also use in your own application, with the ability to override some attributes for your own needs. You can use extends on any service together with other configuration keys. The extends value must be a mapping defined with a required service and an optional file key. extends: file: common.yml service: webapp service : Defines the name of the service being referenced as a base, for example web or database . file : The location of a Compose configuration file defining that service.","title":"extends"},{"location":"docker/#restrictions","text":"The following restrictions apply to the service being referenced: Services that have dependencies on other services cannot be used as a base. Therefore, any key that introduces a dependency on another service is incompatible with extends . The non-exhaustive list of such keys is: links , volumes_from , container mode (in ipc , pid , network_mode and net ), service mode (in ipc , pid and network_mode ), depends_on . Services cannot have circular references with extends . Compose returns an error in all of these cases.","title":"Restrictions"},{"location":"docker/#finding-referenced-service","text":"file value can be: Not present. This indicates that another service within the same Compose file is being referenced. File path, which can be either: Relative path. This path is considered as relative to the location of the main Compose file. Absolute path. A service denoted by service must be present in the identified referenced Compose file. Compose returns an error if: The service denoted by service is not found. The Compose file denoted by file is not found.","title":"Finding referenced service"},{"location":"docker/#merging-service-definitions","text":"Two service definitions, the main one in the current Compose file and the referenced one specified by extends , are merged in the following way: Mappings: Keys in mappings of the main service definition override keys in mappings of the referenced service definition. Keys that aren't overridden are included as is. Sequences: Items are combined together into a new sequence. The order of elements is preserved with the referenced items coming first and main items after. Scalars: Keys in the main service definition take precedence over keys in the referenced one.","title":"Merging service definitions"},{"location":"docker/#mappings","text":"The following keys should be treated as mappings: annotations , build.args , build.labels , build.extra_hosts , deploy.labels , deploy.update_config , deploy.rollback_config , deploy.restart_policy , deploy.resources.limits , environment , healthcheck , labels , logging.options , sysctls , storage_opt , extra_hosts , ulimits . One exception that applies to healthcheck is that the main mapping cannot specify disable: true unless the referenced mapping also specifies disable: true . Compose returns an error in this case. For example, the input below: services: common: image: busybox environment: TZ: utc PORT: 80 cli: extends: service: common environment: PORT: 8080 Produces the following configuration for the cli service. The same output is produced if array syntax is used. environment: PORT: 8080 TZ: utc image: busybox Items under blkio_config.device_read_bps , blkio_config.device_read_iops , blkio_config.device_write_bps , blkio_config.device_write_iops , devices and volumes are also treated as mappings where key is the target path inside the container. For example, the input below: services: common: image: busybox volumes: - common-volume:/var/lib/backup/data:rw cli: extends: service: common volumes: - cli-volume:/var/lib/backup/data:ro Produces the following configuration for the cli service. Note that the mounted path now points to the new volume name and ro flag was applied. image: busybox volumes: - cli-volume:/var/lib/backup/data:ro If the referenced service definition contains extends mapping, the items under it are simply copied into the new merged definition. The merging process is then kicked off again until no extends keys are remaining. For example, the input below: services: base: image: busybox user: root common: image: busybox extends: service: base cli: extends: service: common Produces the following configuration for the cli service. Here, cli services gets user key from common service, which in turn gets this key from base service. image: busybox user: root","title":"Mappings"},{"location":"docker/#sequences","text":"The following keys should be treated as sequences: cap_add , cap_drop , configs , deploy.placement.constraints , deploy.placement.preferences , deploy.reservations.generic_resources , device_cgroup_rules , expose , external_links , ports , secrets , security_opt . Any duplicates resulting from the merge are removed so that the sequence only contains unique elements. For example, the input below: services: common: image: busybox security_opt: - label:role:ROLE cli: extends: service: common security_opt: - label:user:USER Produces the following configuration for the cli service. image: busybox security_opt: - label:role:ROLE - label:user:USER In case list syntax is used, the following keys should also be treated as sequences: dns , dns_search , env_file , tmpfs . Unlike sequence fields mentioned above, duplicates resulting from the merge are not removed.","title":"Sequences"},{"location":"docker/#scalars","text":"Any other allowed keys in the service definition should be treated as scalars.","title":"Scalars"},{"location":"docker/#annotations","text":"annotations defines annotations for the container. annotations can use either an array or a map. annotations: com.example.foo: bar annotations: - com.example.foo=bar","title":"annotations"},{"location":"docker/#external_links","text":"external_links link service containers to services managed outside of your Compose application. external_links define the name of an existing service to retrieve using the platform lookup mechanism. An alias of the form SERVICE:ALIAS can be specified. external_links: - redis - database:mysql - database:postgresql","title":"external_links"},{"location":"docker/#extra_hosts","text":"extra_hosts adds hostname mappings to the container network interface configuration ( /etc/hosts for Linux).","title":"extra_hosts"},{"location":"docker/#short-syntax_2","text":"Short syntax uses plain strings in a list. Values must set hostname and IP address for additional hosts in the form of HOSTNAME=IP . extra_hosts: - \"somehost=162.242.195.82\" - \"otherhost=50.31.209.229\" - \"myhostv6=::1\" IPv6 addresses can be enclosed in square brackets, for example: extra_hosts: - \"myhostv6=[::1]\" The separator = is preferred, but : can also be used. For example: extra_hosts: - \"somehost:162.242.195.82\" - \"myhostv6:::1\"","title":"Short syntax"},{"location":"docker/#long-syntax_2","text":"Alternatively, extra_hosts can be set as a mapping between hostname(s) and IP(s) extra_hosts: somehost: \"162.242.195.82\" otherhost: \"50.31.209.229\" myhostv6: \"::1\" Compose creates a matching entry with the IP address and hostname in the container's network configuration, which means for Linux /etc/hosts get extra lines: 162.242.195.82 somehost 50.31.209.229 otherhost ::1 myhostv6","title":"Long syntax"},{"location":"docker/#group_add","text":"group_add specifies additional groups, by name or number, which the user inside the container must be a member of. An example of where this is useful is when multiple containers (running as different users) need to all read or write the same file on a shared volume. That file can be owned by a group shared by all the containers, and specified in group_add . services: myservice: image: alpine group_add: - mail Running id inside the created container must show that the user belongs to the mail group, which would not have been the case if group_add were not declared.","title":"group_add"},{"location":"docker/#healthcheck","text":"healthcheck declares a check that's run to determine whether or not the service containers are \"healthy\". It works in the same way, and has the same default values, as the HEALTHCHECK Dockerfile instruction set by the service's Docker image. Your Compose file can override the values set in the Dockerfile. healthcheck: test: [\"CMD\", \"curl\", \"-f\", \"http://localhost\"] interval: 1m30s timeout: 10s retries: 3 start_period: 40s start_interval: 5s interval , timeout , start_period , and start_interval are specified as durations . test defines the command Compose runs to check container health. It can be either a string or a list. If it's a list, the first item must be either NONE , CMD or CMD-SHELL . If it's a string, it's equivalent to specifying CMD-SHELL followed by that string. ## Hit the local web app test: [\"CMD\", \"curl\", \"-f\", \"http://localhost\"] Using CMD-SHELL runs the command configured as a string using the container's default shell ( /bin/sh for Linux). Both forms below are equivalent: test: [\"CMD-SHELL\", \"curl -f http://localhost || exit 1\"] test: curl -f https://localhost || exit 1 NONE disables the healthcheck, and is mostly useful to disable the Healthcheck Dockerfile instruction set by the service's Docker image. Alternatively, the healthcheck set by the image can be disabled by setting disable: true : healthcheck: disable: true","title":"healthcheck"},{"location":"docker/#hostname","text":"hostname declares a custom host name to use for the service container. It must be a valid RFC 1123 hostname.","title":"hostname"},{"location":"docker/#image","text":"image specifies the image to start the container from. image must follow the Open Container Specification addressable image format , as [/][/][:|@] . image: redis image: redis:5 image: redis@sha256:0ed5d5928d4737458944eb604cc8509e245c3e19d02ad83935398bc4b991aac7 image: library/redis image: docker.io/library/redis image: my_private.registry:5000/redis If the image does not exist on the platform, Compose attempts to pull it based on the pull_policy . If you are also using the Compose Build Specification , there are alternative options for controlling the precedence of pull over building the image from source, however pulling the image is the default behavior. image may be omitted from a Compose file as long as a build section is declared. If you are not using the Compose Build Specification, Compose won't work if image is missing from the Compose file.","title":"image"},{"location":"docker/#init","text":"init runs an init process (PID 1) inside the container that forwards signals and reaps processes. Set this option to true to enable this feature for the service. services: web: image: alpine:latest init: true The init binary that is used is platform specific.","title":"init"},{"location":"docker/#ipc","text":"ipc configures the IPC isolation mode set by the service container. Available values are platform specific, but Compose defines specific values which must be implemented as described if supported: shareable : Gives the container its own private IPC namespace, with a possibility to share it with other containers. service:{name} : Makes the container join another container's ( shareable ) IPC namespace. ipc: \"shareable\" ipc: \"service:[service name]\"","title":"ipc"},{"location":"docker/#uts","text":"uts configures the UTS namespace mode set for the service container. When unspecified it is the runtime's decision to assign a UTS namespace, if supported. Available values are: 'host' : Results in the container using the same UTS namespace as the host. uts: \"host\"","title":"uts"},{"location":"docker/#isolation","text":"isolation specifies a container\u2019s isolation technology. Supported values are platform specific.","title":"isolation"},{"location":"docker/#labels","text":"labels add metadata to containers. You can use either an array or a map. It's recommended that you use reverse-DNS notation to prevent your labels from conflicting with those used by other software. labels: com.example.description: \"Accounting webapp\" com.example.department: \"Finance\" com.example.label-with-empty-value: \"\" labels: - \"com.example.description=Accounting webapp\" - \"com.example.department=Finance\" - \"com.example.label-with-empty-value\" Compose creates containers with canonical labels: com.docker.compose.project set on all resources created by Compose to the user project name com.docker.compose.service set on service containers with service name as defined in the Compose file The com.docker.compose label prefix is reserved. Specifying labels with this prefix in the Compose file results in a runtime error.","title":"labels"},{"location":"docker/#links","text":"Note Availability of the links attribute is implementation specific. links defines a network link to containers in another service. Either specify both the service name and a link alias ( SERVICE:ALIAS ), or just the service name. web: links: - db - db:database - redis Containers for the linked service are reachable at a hostname identical to the alias, or the service name if no alias is specified. Links are not required to enable services to communicate. When no specific network configuration is set, any service is able to reach any other service at that service\u2019s name on the default network. If services do declare networks they are attached to, links does not override the network configuration and services not attached to a shared network are not be able to communicate. Compose doesn't warn you about a configuration mismatch. Links also express implicit dependency between services in the same way as depends_on , so they determine the order of service startup.","title":"links"},{"location":"docker/#logging","text":"logging defines the logging configuration for the service. logging: driver: syslog options: syslog-address: \"tcp://192.168.0.42:123\" The driver name specifies a logging driver for the service's containers. The default and available values are platform specific. Driver specific options can be set with options as key-value pairs.","title":"logging"},{"location":"docker/#network_mode","text":"network_mode sets a service container's network mode. Available values are platform specific, but Compose defines specific values which must be implemented as described if supported: none : Turns off all container networking. host : Gives the container raw access to the host's network interface. service:{name} : Gives the containers access to the specified service only. network_mode: \"host\" network_mode: \"none\" network_mode: \"service:[service name]\" When set, the networks attribute is not allowed and Compose rejects any Compose file containing both attributes.","title":"network_mode"},{"location":"docker/#networks","text":"networks defines the networks that service containers are attached to, referencing entries under the top-level networks key . services: some-service: networks: - some-network - other-network","title":"networks"},{"location":"docker/#aliases","text":"aliases declares alternative hostnames for the service on the network. Other containers on the same network can use either the service name or an alias to connect to one of the service's containers. Since aliases are network-scoped, the same service can have different aliases on different networks. Note A network-wide alias can be shared by multiple containers, and even by multiple services. If it is, then exactly which container the name resolves to is not guaranteed. services: some-service: networks: some-network: aliases: - alias1 - alias3 other-network: aliases: - alias2 In the following example, service frontend is able to reach the backend service at the hostname backend or database on the back-tier network. The service monitoring is able to reach same backend service at backend or mysql on the admin network. services: frontend: image: example/webapp networks: - front-tier - back-tier monitoring: image: example/monitoring networks: - admin backend: image: example/backend networks: back-tier: aliases: - database admin: aliases: - mysql networks: front-tier: back-tier: admin:","title":"aliases"},{"location":"docker/#ipv4_address-ipv6_address","text":"Specify a static IP address for a service container when joining the network. The corresponding network configuration in the top-level networks section must have an ipam attribute with subnet configurations covering each static address. services: frontend: image: example/webapp networks: front-tier: ipv4_address: 172.16.238.10 ipv6_address: 2001:3984:3989::10 networks: front-tier: ipam: driver: default config: - subnet: \"172.16.238.0/24\" - subnet: \"2001:3984:3989::/64\"","title":"ipv4_address, ipv6_address"},{"location":"docker/#link_local_ips","text":"link_local_ips specifies a list of link-local IPs. Link-local IPs are special IPs which belong to a well known subnet and are purely managed by the operator, usually dependent on the architecture where they are deployed. Implementation is platform specific. Example: services: app: image: busybox command: top networks: app_net: link_local_ips: - 57.123.22.11 - 57.123.22.13 networks: app_net: driver: bridge","title":"link_local_ips"},{"location":"docker/#mac_address","text":"mac_address sets the MAC address used by the service container when connecting to this particular network.","title":"mac_address"},{"location":"docker/#priority","text":"priority indicates in which order Compose connects the service\u2019s containers to its networks. If unspecified, the default value is 0. In the following example, the app service connects to app_net_1 first as it has the highest priority. It then connects to app_net_3 , then app_net_2 , which uses the default priority value of 0. services: app: image: busybox command: top networks: app_net_1: priority: 1000 app_net_2: app_net_3: priority: 100 networks: app_net_1: app_net_2: app_net_3:","title":"priority"},{"location":"docker/#mac_address_1","text":"mac_address sets a MAC address for the service container. Note Container runtimes might reject this value (ie. Docker Engine >= v25.0). In that case, you should use networks.mac_address instead.","title":"mac_address"},{"location":"docker/#mem_limit","text":"DEPRECATED: use deploy.limits.memory","title":"mem_limit"},{"location":"docker/#mem_reservation","text":"DEPRECATED: use deploy.reservations.memory","title":"mem_reservation"},{"location":"docker/#mem_swappiness","text":"mem_swappiness defines as a percentage, a value between 0 and 100, for the host kernel to swap out anonymous memory pages used by a container. 0 : Turns off anonymous page swapping. 100 : Sets all anonymous pages as swappable. The default value is platform specific.","title":"mem_swappiness"},{"location":"docker/#memswap_limit","text":"memswap_limit defines the amount of memory the container is allowed to swap to disk. This is a modifier attribute that only has meaning if memory is also set. Using swap lets the container write excess memory requirements to disk when the container has exhausted all the memory that is available to it. There is a performance penalty for applications that swap memory to disk often. If memswap_limit is set to a positive integer, then both memory and memswap_limit must be set. memswap_limit represents the total amount of memory and swap that can be used, and memory controls the amount used by non-swap memory. So if memory =\"300m\" and memswap_limit =\"1g\", the container can use 300m of memory and 700m (1g - 300m) swap. If memswap_limit is set to 0, the setting is ignored, and the value is treated as unset. If memswap_limit is set to the same value as memory , and memory is set to a positive integer, the container does not have access to swap. If memswap_limit is unset, and memory is set, the container can use as much swap as the memory setting, if the host container has swap memory configured. For instance, if memory =\"300m\" and memswap_limit is not set, the container can use 600m in total of memory and swap. If memswap_limit is explicitly set to -1, the container is allowed to use unlimited swap, up to the amount available on the host system.","title":"memswap_limit"},{"location":"docker/#oom_kill_disable","text":"If oom_kill_disable is set, Compose configures the platform so it won't kill the container in case of memory starvation.","title":"oom_kill_disable"},{"location":"docker/#oom_score_adj","text":"oom_score_adj tunes the preference for containers to be killed by platform in case of memory starvation. Value must be within [-1000,1000] range.","title":"oom_score_adj"},{"location":"docker/#pid","text":"pid sets the PID mode for container created by Compose. Supported values are platform specific.","title":"pid"},{"location":"docker/#pids_limit","text":"DEPRECATED: use deploy.resources.limits.pids pids_limit tunes a container\u2019s PIDs limit. Set to -1 for unlimited PIDs. pids_limit: 10","title":"pids_limit"},{"location":"docker/#platform","text":"platform defines the target platform the containers for the service run on. It uses the os[/arch[/variant]] syntax. The values of os , arch , and variant must conform to the convention used by the OCI Image Spec . Compose uses this attribute to determine which version of the image is pulled and/or on which platform the service\u2019s build is performed. platform: darwin platform: windows/amd64 platform: linux/arm64/v8","title":"platform"},{"location":"docker/#ports","text":"Exposes container ports. Note Port mapping must not be used with network_mode: host otherwise a runtime error occurs.","title":"ports"},{"location":"docker/#short-syntax_3","text":"The short syntax is a colon-separated string to set the host IP, host port, and container port in the form: [HOST:]CONTAINER[/PROTOCOL] where: HOST is [IP:](port | range) CONTAINER is port | range PROTOCOL to restrict port to specified protocol. tcp and udp values are defined by the Specification, Compose offers support for platform-specific protocol names. If host IP is not set, it binds to all network interfaces. Ports can be either a single value or a range. Host and container must use equivalent ranges. Either specify both ports ( HOST:CONTAINER ), or just the container port. In the latter case, the container runtime automatically allocates any unassigned port of the host. HOST:CONTAINER should always be specified as a (quoted) string, to avoid conflicts with yaml base-60 float . Examples: ports: - \"3000\" - \"3000-3005\" - \"8000:8000\" - \"9090-9091:8080-8081\" - \"49100:22\" - \"8000-9000:80\" - \"127.0.0.1:8001:8001\" - \"127.0.0.1:5000-5010:5000-5010\" - \"6060:6060/udp\" Note If Host IP mapping is not supported by a container engine, Compose rejects the Compose file and ignores the specified host IP.","title":"Short syntax"},{"location":"docker/#long-syntax_3","text":"The long form syntax allows the configuration of additional fields that can't be expressed in the short form. target : The container port published : The publicly exposed port. It is defined as a string and can be set as a range using syntax start-end . It means the actual port is assigned a remaining available port, within the set range. host_ip : The Host IP mapping, unspecified means all network interfaces ( 0.0.0.0 ). protocol : The port protocol ( tcp or udp ). Defaults to tcp . mode : host : For publishing a host port on each node, or ingress for a port to be load balanced. Defaults to ingress . name : A human-readable name for the port, used to document it's usage within the service ports: - name: http target: 80 host_ip: 127.0.0.1 published: \"8080\" protocol: tcp mode: host - name: https target: 443 host_ip: 127.0.0.1 published: \"8083-9000\" protocol: tcp mode: host","title":"Long syntax"},{"location":"docker/#privileged","text":"privileged configures the service container to run with elevated privileges. Support and actual impacts are platform specific.","title":"privileged"},{"location":"docker/#profiles","text":"profiles defines a list of named profiles for the service to be enabled under. If unassigned, the service is always started but if assigned, it is only started if the profile is activated. If present, profiles follow the regex format of [a-zA-Z0-9][a-zA-Z0-9_.-]+ . services: frontend: image: frontend profiles: [\"frontend\"] phpmyadmin: image: phpmyadmin depends_on: - db profiles: - debug","title":"profiles"},{"location":"docker/#pull_policy","text":"pull_policy defines the decisions Compose makes when it starts to pull images. Possible values are: always : Compose always pulls the image from the registry. never : Compose doesn't pull the image from a registry and relies on the platform cached image. If there is no cached image, a failure is reported. missing : Compose pulls the image only if it's not available in the platform cache. This is the default option if you are not also using the Compose Build Specification . if_not_present is considered an alias for this value for backward compatibility. build : Compose builds the image. Compose rebuilds the image if it's already present.","title":"pull_policy"},{"location":"docker/#read_only","text":"read_only configures the service container to be created with a read-only filesystem.","title":"read_only"},{"location":"docker/#restart","text":"restart defines the policy that the platform applies on container termination. no : The default restart policy. It does not restart the container under any circumstances. always : The policy always restarts the container until its removal. on-failure : The policy restarts the container if the exit code indicates an error. unless-stopped : The policy restarts the container irrespective of the exit code but stops restarting when the service is stopped or removed. restart: \"no\" restart: always restart: on-failure restart: unless-stopped","title":"restart"},{"location":"docker/#runtime","text":"runtime specifies which runtime to use for the service\u2019s containers. The value of runtime is specific to the implementation. For example, runtime can be the name of an implementation of OCI Runtime Spec , such as \"runc\". web: image: busybox:latest command: true runtime: runc","title":"runtime"},{"location":"docker/#scale","text":"scale specifies the default number of containers to deploy for this service. When both are set, scale must be consistent with the replicas attribute in the Deploy Specification .","title":"scale"},{"location":"docker/#secrets","text":"secrets grants access to sensitive data defined by secrets on a per-service basis. Two different syntax variants are supported; the short syntax and the long syntax. Compose reports an error if the secret doesn't exist on the platform or isn't defined in the secrets section of the Compose file. Services can be granted access to multiple secrets. Long and short syntax for secrets may be used in the same Compose file. Defining a secret in the top-level secrets must not imply granting any service access to it. Such grant must be explicit within service specification as secrets service element.","title":"secrets"},{"location":"docker/#short-syntax_4","text":"The short syntax variant only specifies the secret name. This grants the container access to the secret and mounts it as read-only to /run/secrets/ within the container. The source name and destination mountpoint are both set to the secret name. The following example uses the short syntax to grant the frontend service access to the server-certificate secret. The value of server-certificate is set to the contents of the file ./server.cert . services: frontend: image: example/webapp secrets: - server-certificate secrets: server-certificate: file: ./server.cert","title":"Short syntax"},{"location":"docker/#long-syntax_4","text":"The long syntax provides more granularity in how the secret is created within the service's containers. source : The name of the secret as it exists on the platform. target : The name of the file to be mounted in /run/secrets/ in the service's task container, or absolute path of the file if an alternate location is required. Defaults to source if not specified. uid and gid : The numeric UID or GID that owns the file within /run/secrets/ in the service's task containers. Default value is USER running container. mode : The permissions for the file to be mounted in /run/secrets/ in the service's task containers, in octal notation. The default value is world-readable permissions (mode 0444 ). The writable bit must be ignored if set. The executable bit may be set. Note that the uid , gid , and mode attributes are implementation specific. The following example sets the name of the server-certificate secret file to server.crt within the container, sets the mode to 0440 (group-readable), and sets the user and group to 103 . The value of server-certificate secret is provided by the platform through a lookup and the secret's lifecycle is not directly managed by Compose. services: frontend: image: example/webapp secrets: - source: server-certificate target: server.cert uid: \"103\" gid: \"103\" mode: 0440 secrets: server-certificate: external: true","title":"Long syntax"},{"location":"docker/#security_opt","text":"security_opt overrides the default labeling scheme for each container. security_opt: - label:user:USER - label:role:ROLE For further default labeling schemes you can override, see Security configuration .","title":"security_opt"},{"location":"docker/#shm_size","text":"shm_size configures the size of the shared memory ( /dev/shm partition on Linux) allowed by the service container. It's specified as a byte value .","title":"shm_size"},{"location":"docker/#stdin_open","text":"stdin_open configures a service containers to run with an allocated stdin.","title":"stdin_open"},{"location":"docker/#stop_grace_period","text":"stop_grace_period specifies how long Compose must wait when attempting to stop a container if it doesn't handle SIGTERM (or whichever stop signal has been specified with stop_signal ), before sending SIGKILL. It's specified as a duration . stop_grace_period: 1s stop_grace_period: 1m30s Default value is 10 seconds for the container to exit before sending SIGKILL.","title":"stop_grace_period"},{"location":"docker/#stop_signal","text":"stop_signal defines the signal that Compose uses to stop the service containers. If unset containers are stopped by Compose by sending SIGTERM . stop_signal: SIGUSR1","title":"stop_signal"},{"location":"docker/#storage_opt","text":"storage_opt defines storage driver options for a service. storage_opt: size: '1G'","title":"storage_opt"},{"location":"docker/#sysctls","text":"sysctls defines kernel parameters to set in the container. sysctls can use either an array or a map. sysctls: net.core.somaxconn: 1024 net.ipv4.tcp_syncookies: 0 sysctls: - net.core.somaxconn=1024 - net.ipv4.tcp_syncookies=0 You can only use sysctls that are namespaced in the kernel. Docker does not support changing sysctls inside a container that also modify the host system. For an overview of supported sysctls, refer to configure namespaced kernel parameters (sysctls) at runtime .","title":"sysctls"},{"location":"docker/#tmpfs","text":"tmpfs mounts a temporary file system inside the container. It can be a single value or a list. tmpfs: /run tmpfs: - /run - /tmp","title":"tmpfs"},{"location":"docker/#tty","text":"tty configures service container to run with a TTY.","title":"tty"},{"location":"docker/#ulimits","text":"ulimits overrides the default ulimits for a container. It's specified either as an integer for a single limit or as mapping for soft/hard limits. ulimits: nproc: 65535 nofile: soft: 20000 hard: 40000","title":"ulimits"},{"location":"docker/#user","text":"user overrides the user used to run the container process. The default is set by the image (i.e. Dockerfile USER ). If it's not set, then root .","title":"user"},{"location":"docker/#userns_mode","text":"userns_mode sets the user namespace for the service. Supported values are platform specific and may depend on platform configuration. userns_mode: \"host\"","title":"userns_mode"},{"location":"docker/#volumes","text":"volumes define mount host paths or named volumes that are accessible by service containers. You can use volumes to define multiple types of mounts; volume , bind , tmpfs , or npipe . If the mount is a host path and is only used by a single service, it can be declared as part of the service definition. To reuse a volume across multiple services, a named volume must be declared in the top-level volumes key . The following example shows a named volume ( db-data ) being used by the backend service, and a bind mount defined for a single service. services: backend: image: example/backend volumes: - type: volume source: db-data target: /data volume: nocopy: true - type: bind source: /var/run/postgres/postgres.sock target: /var/run/postgres/postgres.sock volumes: db-data:","title":"volumes"},{"location":"docker/#short-syntax_5","text":"The short syntax uses a single string with colon-separated values to specify a volume mount ( VOLUME:CONTAINER_PATH ), or an access mode ( VOLUME:CONTAINER_PATH:ACCESS_MODE ). VOLUME : Can be either a host path on the platform hosting containers (bind mount) or a volume name. CONTAINER_PATH : The path in the container where the volume is mounted. ACCESS_MODE : A comma-separated , list of options: rw : Read and write access. This is the default if none is specified. ro : Read-only access. z : SELinux option indicating that the bind mount host content is shared among multiple containers. Z : SELinux option indicating that the bind mount host content is private and unshared for other containers. Note The SELinux re-labeling bind mount option is ignored on platforms without SELinux. Note Relative host paths are only supported by Compose that deploy to a local container runtime. This is because the relative path is resolved from the Compose file\u2019s parent directory which is only applicable in the local case. When Compose deploys to a non-local platform it rejects Compose files which use relative host paths with an error. To avoid ambiguities with named volumes, relative paths should always begin with . or .. .","title":"Short syntax"},{"location":"docker/#long-syntax_5","text":"The long form syntax allows the configuration of additional fields that can't be expressed in the short form. type : The mount type. Either volume , bind , tmpfs , npipe , or cluster source : The source of the mount, a path on the host for a bind mount, or the name of a volume defined in the top-level volumes key . Not applicable for a tmpfs mount. target : The path in the container where the volume is mounted. read_only : Flag to set the volume as read-only. bind : Used to configure additional bind options: propagation : The propagation mode used for the bind. create_host_path : Creates a directory at the source path on host if there is nothing present. Compose does nothing if there is something present at the path. This is automatically implied by short syntax for backward compatibility with docker-compose legacy. selinux : The SELinux re-labeling option z (shared) or Z (private) volume : Configures additional volume options: nocopy : Flag to disable copying of data from a container when a volume is created. tmpfs : Configures additional tmpfs options: size : The size for the tmpfs mount in bytes (either numeric or as bytes unit). mode : The file mode for the tmpfs mount as Unix permission bits as an octal number. consistency : The consistency requirements of the mount. Available values are platform specific.","title":"Long syntax"},{"location":"docker/#volumes_from","text":"volumes_from mounts all of the volumes from another service or container. You can optionally specify read-only access ro or read-write rw . If no access level is specified, then read-write access is used. You can also mount volumes from a container that is not managed by Compose by using the container: prefix. volumes_from: - service_name - service_name:ro - container:container_name - container:container_name:rw","title":"volumes_from"},{"location":"docker/#working_dir","text":"working_dir overrides the container's working directory which is specified by the image, for example Dockerfile's WORKDIR .","title":"working_dir"},{"location":"docker/#networks-top-level-element","text":"Networks are the layer that allow services to communicate with each other. The top-level networks element lets you configure named networks that can be reused across multiple services. To use a network across multiple services, you must explicitly grant each service access by using the networks attribute within the services top-level element. The networks top-level element has additional syntax that provides more granular control.","title":"Networks top-level element"},{"location":"docker/#examples","text":"","title":"Examples"},{"location":"docker/#basic-example","text":"In the following example, at runtime, networks front-tier and back-tier are created and the frontend service is connected to front-tier and back-tier networks. services: frontend: image: example/webapp networks: - front-tier - back-tier networks: front-tier: back-tier:","title":"Basic example"},{"location":"docker/#advanced-example","text":"services: proxy: build: ./proxy networks: - frontend app: build: ./app networks: - frontend - backend db: image: postgres networks: - backend networks: frontend: ## Use a custom driver driver: custom-driver-1 backend: ## Use a custom driver which takes special options driver: custom-driver-2 driver_opts: foo: \"1\" bar: \"2\" The advanced example shows a Compose file which defines two custom networks. The proxy service is isolated from the db service, because they do not share a network in common. Only app can talk to both.","title":"Advanced example"},{"location":"docker/#attributes","text":"","title":"Attributes"},{"location":"docker/#driver","text":"driver specifies which driver should be used for this network. Compose returns an error if the driver is not available on the platform. networks: db-data: driver: overlay Default and available values are platform specific. Compose supports the following drivers: none and host host : Use the host's networking stack. none : Turn off networking.","title":"driver"},{"location":"docker/#host-or-none","text":"The syntax for using built-in networks such as host and none is different, as such networks implicitly exist outside the scope of Compose. To use them, you must define an external network with the name host or none and an alias that Compose can use ( hostnet and nonet in the following example), then grant the service access to that network using its alias. services: web: networks: hostnet: {} networks: hostnet: external: true name: host services: web: ... networks: nonet: {} networks: nonet: external: true name: none","title":"host or none"},{"location":"docker/#driver_opts","text":"driver_opts specifies a list of options as key-value pairs to pass to the driver. These options are driver-dependent. Consult the driver's documentation for more information. networks: db-data: driver_opts: foo: \"bar\" baz: 1","title":"driver_opts"},{"location":"docker/#attachable","text":"If attachable is set to true , then standalone containers should be able to attach to this network, in addition to services. If a standalone container attaches to the network, it can communicate with services and other standalone containers that are also attached to the network. networks: mynet1: driver: overlay attachable: true","title":"attachable"},{"location":"docker/#enable_ipv6","text":"enable_ipv6 enables IPv6 networking. For an example, see step four of Create an IPv6 network .","title":"enable_ipv6"},{"location":"docker/#external","text":"If set to true : - external specifies that this network\u2019s lifecycle is maintained outside of that of the application. Compose doesn't attempt to create these networks, and returns an error if one doesn't exist. - All other attributes apart from name are irrelevant. If Compose detects any other attribute, it rejects the Compose file as invalid. In the example below, proxy is the gateway to the outside world. Instead of attempting to create a network, Compose queries the platform for an existing network simply called outside and connects the proxy service's containers to it. services: proxy: image: example/proxy networks: - outside - default app: image: example/app networks: - default networks: outside: external: true","title":"external"},{"location":"docker/#ipam","text":"ipam specifies a custom IPAM configuration. This is an object with several properties, each of which is optional: driver : Custom IPAM driver, instead of the default. config : A list with zero or more configuration elements, each containing a: subnet : Subnet in CIDR format that represents a network segment ip_range : Range of IPs from which to allocate container IPs gateway : IPv4 or IPv6 gateway for the master subnet aux_addresses : Auxiliary IPv4 or IPv6 addresses used by Network driver, as a mapping from hostname to IP options : Driver-specific options as a key-value mapping. networks: mynet1: ipam: driver: default config: - subnet: 172.28.0.0/16 ip_range: 172.28.5.0/24 gateway: 172.28.5.254 aux_addresses: host1: 172.28.1.5 host2: 172.28.1.6 host3: 172.28.1.7 options: foo: bar baz: \"0\"","title":"ipam"},{"location":"docker/#internal","text":"By default, Compose provides external connectivity to networks. internal , when set to true , allows you to create an externally isolated network.","title":"internal"},{"location":"docker/#labels_1","text":"Add metadata to containers using labels . You can use either an array or a dictionary. It is recommended that you use reverse-DNS notation to prevent labels from conflicting with those used by other software. networks: mynet1: labels: com.example.description: \"Financial transaction network\" com.example.department: \"Finance\" com.example.label-with-empty-value: \"\" networks: mynet1: labels: - \"com.example.description=Financial transaction network\" - \"com.example.department=Finance\" - \"com.example.label-with-empty-value\" Compose sets com.docker.compose.project and com.docker.compose.network labels.","title":"labels"},{"location":"docker/#name","text":"name sets a custom name for the network. The name field can be used to reference networks which contain special characters. The name is used as is and is not scoped with the project name. networks: network1: name: my-app-net It can also be used in conjunction with the external property to define the platform network that Compose should retrieve, typically by using a parameter so the Compose file doesn't need to hard-code runtime specific values: networks: network1: external: true name: \"${NETWORK_ID}\"","title":"name"},{"location":"docker/#volumes-top-level-element","text":"Volumes are persistent data stores implemented by the container engine. Compose offers a neutral way for services to mount volumes, and configuration parameters to allocate them to infrastructure. The top-level volumes declaration lets you configure named volumes that can be reused across multiple services. To use a volume across multiple services, you must explicitly grant each service access by using the volumes attribute within the services top-level element. The volumes attribute has additional syntax that provides more granular control.","title":"Volumes top-level element"},{"location":"docker/#example","text":"The following example shows a two-service setup where a database's data directory is shared with another service as a volume, named db-data , so that it can be periodically backed up. services: backend: image: example/database volumes: - db-data:/etc/data backup: image: backup-service volumes: - db-data:/var/lib/backup/data volumes: db-data: The db-data volume is mounted at the /var/lib/backup/data and /etc/data container paths for backup and backend respectively. Running docker compose up creates the volume if it doesn't already exist. Otherwise, the existing volume is used and is recreated if it's manually deleted outside of Compose.","title":"Example"},{"location":"docker/#attributes_1","text":"An entry under the top-level volumes section can be empty, in which case it uses the container engine's default configuration for creating a volume. Optionally, you can configure it with the following keys:","title":"Attributes"},{"location":"docker/#driver_1","text":"Specifies which volume driver should be used. Default and available values are platform specific. If the driver is not available, Compose returns an error and doesn't deploy the application. volumes: db-data: driver: foobar","title":"driver"},{"location":"docker/#driver_opts_1","text":"driver_opts specifies a list of options as key-value pairs to pass to the driver for this volume. The options are driver-dependent. volumes: example: driver_opts: type: \"nfs\" o: \"addr=10.40.0.199,nolock,soft,rw\" device: \":/docker/example\"","title":"driver_opts"},{"location":"docker/#external_1","text":"If set to true : - external specifies that this volume already exists on the platform and its lifecycle is managed outside of that of the application. Compose doesn't then create the volume, and returns an error if the volume doesn't exist. - All other attributes apart from name are irrelevant. If Compose detects any other attribute, it rejects the Compose file as invalid. In the example below, instead of attempting to create a volume called {project_name}_db-data , Compose looks for an existing volume simply called db-data and mounts it into the backend service's containers. services: backend: image: example/database volumes: - db-data:/etc/data volumes: db-data: external: true","title":"external"},{"location":"docker/#labels_2","text":"labels are used to add metadata to volumes. You can use either an array or a dictionary. It's recommended that you use reverse-DNS notation to prevent your labels from conflicting with those used by other software. volumes: db-data: labels: com.example.description: \"Database volume\" com.example.department: \"IT/Ops\" com.example.label-with-empty-value: \"\" volumes: db-data: labels: - \"com.example.description=Database volume\" - \"com.example.department=IT/Ops\" - \"com.example.label-with-empty-value\" Compose sets com.docker.compose.project and com.docker.compose.volume labels.","title":"labels"},{"location":"docker/#name_1","text":"name sets a custom name for a volume. The name field can be used to reference volumes that contain special characters. The name is used as is and is not scoped with the stack name. volumes: db-data: name: \"my-app-data\" This makes it possible to make this lookup name a parameter of the Compose file, so that the model ID for the volume is hard-coded but the actual volume ID on the platform is set at runtime during deployment. For example, if DATABASE_VOLUME=my_volume_001 in your .env file: volumes: db-data: name: ${DATABASE_VOLUME} Running docker compose up uses the volume called my_volume_001 . It can also be used in conjunction with the external property. This means the name of the volume used to lookup the actual volume on the platform is set separately from the name used to refer to it within the Compose file: volumes: db-data: external: name: actual-name-of-volume","title":"name"},{"location":"docker/#configs-top-level-element","text":"Configs allow services to adapt their behaviour without the need to rebuild a Docker image. Services can only access configs when explicitly granted by a configs attribute within the services top-level element. As with volumes, configs are mounted as files into a service's container's filesystem. The location of the mount point within the container defaults to / in Linux containers and C:\\ in Windows containers. By default, the config: - Is owned by the user running the container command but can be overridden by service configuration. - Has world-readable permissions (mode 0444), unless the service is configured to override this. The top-level configs declaration defines or references configuration data that is granted to services in your Compose application. The source of the config is either file or external . file : The config is created with the contents of the file at the specified path. environment : The config content is created with the value of an environment variable. content : The content is created with the inlined value. external : If set to true, external specifies that this config has already been created. Compose does not attempt to create it, and if it does not exist, an error occurs. name : The name of the config object in the container engine to look up. This field can be used to reference configs that contain special characters. The name is used as is and will not be scoped with the project name.","title":"Configs top-level element"},{"location":"docker/#example-1","text":"_http_config is created when the application is deployed, by registering the content of the httpd.conf as the configuration data. configs: http_config: file: ./httpd.conf Alternatively, http_config can be declared as external. Compose looks up http_config to expose the configuration data to relevant services. configs: http_config: external: true","title":"Example 1"},{"location":"docker/#example-2","text":"_app_config is created when the application is deployed, by registering the inlined content as the configuration data. This comes with the benefits Compose will infer variables when creating the config, which allows to adjust content according to service configuration: configs: app_config: content: | debug=${DEBUG} spring.application.admin.enabled=${DEBUG} spring.application.name=${COMPOSE_PROJECT_NAME}","title":"Example 2"},{"location":"docker/#example-3","text":"External configs lookup can also use a distinct key by specifying a name . The following example modifies the previous one to look up a config using the parameter HTTP_CONFIG_KEY . The the actual lookup key will is set at deployment time by the interpolation of variables, but exposed to containers as hard-coded ID http_config . configs: http_config: external: true name: \"${HTTP_CONFIG_KEY}\" If external is set to true , all other attributes apart from name are irrelevant. If Compose detecs any other attribute, it rejects the Compose file as invalid.","title":"Example 3"},{"location":"docker/#secrets-top-level-element","text":"Secrets are a flavor of Configs focusing on sensitive data, with specific constraint for this usage. Services can only access secrets when explicitly granted by a secrets attribute within the services top-level element. The top-level secrets declaration defines or references sensitive data that is granted to the services in your Compose application. The source of the secret is either file or environment . file : The secret is created with the contents of the file at the specified path. environment : The secret is created with the value of an environment variable. external : If set to true, external specifies that this secret has already been created. Compose does not attempt to create it, and if it does not exist, an error occurs. name : The name of the secret object in Docker. This field can be used to reference secrets that contain special characters. The name is used as is and isn't scoped with the project name.","title":"Secrets top-level element"},{"location":"docker/#example-1_1","text":"server-certificate secret is created as _server-certificate when the application is deployed, by registering content of the server.cert as a platform secret. secrets: server-certificate: file: ./server.cert","title":"Example 1"},{"location":"docker/#example-2_1","text":"token secret is created as _token when the application is deployed, by registering the content of the OAUTH_TOKEN environment variable as a platform secret. secrets: token: environment: \"OAUTH_TOKEN\" Alternatively, server-certificate can be declared as external. Compose looks up the server-certificate secret to expose to relevant services. secrets: server-certificate: external: true","title":"Example 2"},{"location":"docker/#example-3_1","text":"External secrets lookup can also use a distinct key by specifying a name . The following example modifies the previous example to look up a secret using the name CERTIFICATE_KEY . The actual lookup key is set at deployment time by the interpolation of variables, but exposed to containers as hard-coded ID server-certificate . secrets: server-certificate: external: true name: \"${CERTIFICATE_KEY}\" If external is set to true , all other attributes apart from name are irrelevant. If Compose detects any other attribute, it rejects the Compose file as invalid. Your Compose file needs to explicitly grant access to the secrets to relevant services in your application.","title":"Example 3"},{"location":"docker/#fragments","text":"With Compose, you can use built-in YAML features to make your Compose file neater and more efficient. Anchors and aliases let you create re-usable blocks. This is useful if you start to find common configurations that span multiple services. Having re-usable blocks minimizes potential mistakes. Anchors are created using the & sign. The sign is followed by an alias name. You can use this alias with the * sign later to reference the value following the anchor. Make sure there is no space between the & and the * characters and the following alias name. You can use more than one anchor and alias in a single Compose file.","title":"Fragments"},{"location":"docker/#example-1_2","text":"volumes: db-data: &default-volume driver: default metrics: *default-volume In the example above, a default-volume anchor is created based on the db-data volume. It is later reused by the alias *default-volume to define the metrics volume. Anchor resolution takes place before variables interpolation , so variables can't be used to set anchors or aliases.","title":"Example 1"},{"location":"docker/#example-2_2","text":"services: first: image: my-image:latest environment: &env - CONFIG_KEY - EXAMPLE_KEY - DEMO_VAR second: image: another-image:latest environment: *env If you have an anchor that you want to use in more than one service, use it in conjunction with an extension to make your Compose file easier to maintain.","title":"Example 2"},{"location":"docker/#example-3_2","text":"You may want to partially override values. Compose follows the rule outlined by YAML merge type . In the following example, metrics volume specification uses alias to avoid repetition but overrides name attribute: services: backend: image: example/database volumes: - db-data - metrics volumes: db-data: &default-volume driver: default name: \"data\" metrics: <<: *default-volume name: \"metrics\"","title":"Example 3"},{"location":"docker/#example-4","text":"You can also extend the anchor to add additional values. services: first: image: my-image:latest environment: &env FOO: BAR ZOT: QUIX second: image: another-image:latest environment: <<: *env YET_ANOTHER: VARIABLE Note YAML merge only applies to mappings, and can't be used with sequences. In example above, the environment variables must be declared using the FOO: BAR mapping syntax, while the sequence syntax - FOO=BAR is only valid when no fragments are involved.","title":"Example 4"},{"location":"docker/#extension","text":"As with Fragments , Extensions can be used to make your Compose file more efficient and easier to maintain. Extensions can also be used with anchors and aliases . Use the prefix x- as a top-level element to modularize configurations that you want to reuse. Compose ignores any fields that start with x- , this is the sole exception where Compose silently ignores unrecognized fields. They also can be used within any structure in a Compose file where user-defined keys are not expected. Compose use those to enable experimental features, the same way browsers add support for custom CSS features","title":"Extension"},{"location":"docker/#example-1_3","text":"x-custom: foo: - bar - zot services: webapp: image: example/webapp x-foo: bar service: backend: deploy: placement: x-aws-role: \"arn:aws:iam::XXXXXXXXXXXX:role/foo\" x-aws-region: \"eu-west-3\" x-azure-region: \"france-central\"","title":"Example 1"},{"location":"docker/#example-2_3","text":"x-env: &env environment: - CONFIG_KEY - EXAMPLE_KEY services: first: <<: *env image: my-image:latest second: <<: *env image: another-image:latest In this example, the environment variables do not belong to either of the services. They\u2019ve been lifted out completely into the x-env extension field. This defines a new node which contains the environment field. The &env YAML anchor is used so both services can reference the extension field\u2019s value as *env .","title":"Example 2"},{"location":"docker/#example-3_3","text":"x-function: &function labels: function: \"true\" depends_on: - gateway networks: - functions deploy: placement: constraints: - 'node.platform.os == linux' services: ## Node.js gives OS info about the node (Host) nodeinfo: <<: *function image: functions/nodeinfo:latest environment: no_proxy: \"gateway\" https_proxy: $https_proxy ## Uses `cat` to echo back response, fastest function to execute. echoit: <<: *function image: functions/alpine:health environment: fprocess: \"cat\" no_proxy: \"gateway\" https_proxy: $https_proxy The nodeinfo and echoit services both include the x-function extension via the &function anchor, then set their specific image and environment.","title":"Example 3"},{"location":"docker/#example-4_1","text":"Using YAML merge it is also possible to use multiple extensions and share and override additional attributes for specific needs: x-environment: &default-environment FOO: BAR ZOT: QUIX x-keys: &keys KEY: VALUE services: frontend: image: example/webapp environment: << : [*default-environment, *keys] YET_ANOTHER: VARIABLE Note YAML merge only applies to mappings, and can't be used with sequences. In the example above, the environment variables are declared using the FOO: BAR mapping syntax, while the sequence syntax - FOO=BAR is only valid when no fragments are involved.","title":"Example 4"},{"location":"docker/#informative-historical-notes","text":"This section is informative. At the time of writing, the following prefixes are known to exist: Prefix Vendor/Organization docker Docker kubernetes Kubernetes","title":"Informative Historical Notes"},{"location":"docker/#specifying-byte-values","text":"Values express a byte value as a string in {amount}{byte unit} format: The supported units are b (bytes), k or kb (kilo bytes), m or mb (mega bytes) and g or gb (giga bytes). 2b 1024kb 2048k 300m 1gb","title":"Specifying byte values"},{"location":"docker/#specifying-durations","text":"Values express a duration as a string in the form of {value}{unit} . The supported units are us (microseconds), ms (milliseconds), s (seconds), m (minutes) and h (hours). Values can combine multiple values without separator. 10ms 40s 1m30s 1h5m30s20ms","title":"Specifying durations"},{"location":"docker/#interpolation","text":"Values in a Compose file can be set by variables and interpolated at runtime. Compose files use a Bash-like syntax ${VARIABLE} . Both $VARIABLE and ${VARIABLE} syntax is supported. Default values can be defined inline using typical shell syntax: ${VARIABLE:-default} evaluates to default if VARIABLE is unset or empty in the environment. ${VARIABLE-default} evaluates to default only if VARIABLE is unset in the environment. Similarly, the following syntax allows you to specify mandatory variables: ${VARIABLE:?err} exits with an error message containing err if VARIABLE is unset or empty in the environment. ${VARIABLE?err} exits with an error message containing err only if VARIABLE is unset in the environment. Interpolation can also be nested: ${VARIABLE:-${FOO}} ${VARIABLE?$FOO} ${VARIABLE:-${FOO:-default}} Other extended shell-style features, such as ${VARIABLE/foo/bar} , are not supported by Compose. You can use a $$ (double-dollar sign) when your configuration needs a literal dollar sign. This also prevents Compose from interpolating a value, so a $$ allows you to refer to environment variables that you don't want processed by Compose. web: build: . command: \"$$VAR_NOT_INTERPOLATED_BY_COMPOSE\" If Compose can't resolve a substituted variable and no default value is defined, it displays a warning and substitutes the variable with an empty string. As any values in a Compose file can be interpolated with variable substitution, including compact string notation for complex elements, interpolation is applied before a merge on a per-file basis. Interpolation applies only to YAML values, not to keys. For the few places where keys are actually arbitrary user-defined strings, such as labels or environment , an alternate equal sign syntax must be used for interpolation to apply. For example: services: foo: labels: \"$VAR_NOT_INTERPOLATED_BY_COMPOSE\": \"BAR\" services: foo: labels: - \"$VAR_INTERPOLATED_BY_COMPOSE=BAR\"","title":"Interpolation"},{"location":"docker/#merge-and-override","text":"Compose lets you define a Compose application model through multiple Compose files . When doing so, Compose follows the rules declared in this section to merge Compose files.","title":"Merge and override"},{"location":"docker/#mapping","text":"A YAML mapping gets merged by adding missing entries and merging the conflicting ones. Merging the following example YAML trees: services: foo: key1: value1 key2: value2 services: foo: key2: VALUE key3: value3 Results in a Compose application model equivalent to the YAML tree: services: foo: key1: value1 key2: VALUE key3: value3","title":"Mapping"},{"location":"docker/#sequence","text":"A YAML sequence is merged by appending values from the overriding Compose file to the previous one. Merging the following example YAML trees: services: foo: DNS: - 1.1.1.1 services: foo: DNS: - 8.8.8.8 Results in a Compose application model equivalent to the YAML tree: services: foo: DNS: - 1.1.1.1 - 8.8.8.8","title":"Sequence"},{"location":"docker/#exceptions","text":"","title":"Exceptions"},{"location":"docker/#shell-commands","text":"When merging Compose files that use the services attributes command , entrypoint and healthcheck: test , the value is overridden by the latest Compose file, and not appended. Merging the following example YAML trees: services: foo: command: [\"echo\", \"foo\"] services: foo: command: [\"echo\", \"bar\"] Results in a Compose application model equivalent to the YAML tree: services: foo: command: [\"echo\", \"bar\"]","title":"Shell commands"},{"location":"docker/#unique-resources","text":"Applies to the ports , volumes , secrets and configs services attributes. While these types are modeled in a Compose file as a sequence, they have special uniqueness requirements: Attribute Unique key volumes target secrets source configs source ports {ip, target, published, protocol} When merging Compose files, Compose appends new entries that do not violate a uniqueness constraint and merge entries that share a unique key. Merging the following example YAML trees: services: foo: volumes: - foo:/work services: foo: volumes: - bar:/work Results in a Compose application model equivalent to the YAML tree: services: foo: volumes: - bar:/work","title":"Unique resources"},{"location":"docker/#reset-value","text":"In addition to the previously described mechanism, an override Compose file can also be used to remove elements from your application model. For this purpose, the custom YAML tag !reset can be set to override a value set by the overriden Compose file. A valid value for attribute must be provided, but will be ignored and target attribute will be set with type's default value or null . For readability, it is recommended to explicitly set the attribute value to the null ( null ) or empty array [] (with !reset null or !reset [] ) so that it is clear that resulting attribute will be cleared. A base compose.yaml file: services: app: image: myapp ports: - \"8080:80\" environment: FOO: BAR And an overide.compose.yaml file: services: app: image: myapp ports: !reset [] environment: FOO: !reset null Results in: services: app: image: myapp","title":"Reset value"},{"location":"docker/#replace-value","text":"While !reset can be used to remove a declaration from a Compose file using an override file, !override allows you to fully replace an attribute, bypassing the standard merge rules. A typical example is to fully replace a resource definition, to rely on a distinct model but using the same name. A base compose.yaml file: services: app: image: myapp ports: - \"8080:80\" To remove the original port, but expose a new one, the following override file is used: services: app: ports: !override - \"8443:443\" This results in: services: app: image: myapp ports: - \"8443:443\" If !override had not been used, both 8080:80 and 8443:443 would be exposed as per the merging rules outlined above .","title":"Replace value"},{"location":"docker/#include","text":"A Compose application can declare dependency on another Compose application. This is useful if: - You want to reuse other Compose files. - You need to factor out parts of your application model into separate Compose files so they can be managed separately or shared with others. - Teams need to keep a Compose file reasonably complicated for the limited amount of resources it has to declare for it's own sub-domain, within a larger deployment. The include top-level section is used to define the dependency on another Compose application, or sub-domain. Each path listed in the include section is loaded as an individual Compose application model, with it's own project directory, in order to resolve relative paths. Once the included Compose application is loaded, all resources definitions are copied into the current Compose application model. Compose displays a warning if resource names conflict and doesn't try to merge them. To enforce this, include is evaluated after the Compose file(s) selected to define the Compose application model have been parsed and merged, so that conflicts between Compose files are detected. include applies recursively so an included Compose file which declares its own include section, triggers those other files to be included as well. Any volumes, networks, or other resources pulled in from the included Compose file can be used by the current Compose application for cross-service references. For example: include: - my-compose-include.yaml #with serviceB declared services: serviceA: build: . depends_on: - serviceB #use serviceB directly as if it was declared in this Compose file Compose also supports the use of interpolated variables with include . It's recommended that you specify mandatory variables . For example: include: -${INCLUDE_PATH:?FOO}/compose.yaml","title":"Include"},{"location":"docker/#short-syntax_6","text":"The short syntax only defines paths to other Compose files. The file is loaded with the parent folder as the project directory, and an optional .env file that is loaded to define any variables' default values by interpolation. The local project's environment can override those values. include: - ../commons/compose.yaml - ../another_domain/compose.yaml services: webapp: depends_on: - included-service ## defined by another_domain In the above example, both ../commons/compose.yaml and ../another_domain/compose.yaml are loaded as individual Compose projects. Relative paths in Compose files being referred by include are resolved relative to their own Compose file path, not based on the local project's directory. Variables are interpolated using values set in the optional .env file in same folder, and is overridden by the local project's environment.","title":"Short syntax"},{"location":"docker/#long-syntax_6","text":"The long syntax offers more control over the sub-project parsing: include: - path: ../commons/compose.yaml project_directory: .. env_file: ../another/.env","title":"Long syntax"},{"location":"docker/#path","text":"path is required and defines the location of the Compose file(s) to be parsed and included into the local Compose model. path can be set either to a string when a single Compose file is involved, or to a list of strings when multiple Compose files need to be merged together to define the Compose model to be included in the local application. include: - path: - ../commons/compose.yaml - ./commons-override.yaml","title":"path"},{"location":"docker/#project_directory","text":"project_directory defines a base path to resolve relative paths set in the Compose file. It defaults to the directory of the included Compose file.","title":"project_directory"},{"location":"docker/#env_file_1","text":"env_file defines an environment file(s) to use to define default values when interpolating variables in the Compose file being parsed. It defaults to .env file in the project_directory for the Compose file being parsed. env_file can be set either to a string or a list of strings when multiple environment files need to be merged to define a project environment. include: - path: ../another/compose.yaml env_file: - ../another/.env - ../another/dev.env The local project's environment has precedence over the values set by the Compose file, so that the local project can override values for customization.","title":"env_file"},{"location":"docker/#profiles_1","text":"With profiles you can define a set of active profiles so your Compose application model is adjusted for various usages and environments. The exact mechanism is implementation specific and may include command line flags, environment variables, etc. The services top-level element supports a profiles attribute to define a list of named profiles. Services without a profiles attribute are always enabled. A service is ignored by Compose when none of the listed profiles match the active ones, unless the service is explicitly targeted by a command. In that case its profile is added to the set of active profiles. Note All other top-level elements are not affected by profiles and are always active. References to other services (by links , extends or shared resource syntax service:xxx ) do not automatically enable a component that would otherwise have been ignored by active profiles. Instead Compose returns an error.","title":"Profiles"},{"location":"docker/#illustrative-example_1","text":"services: foo: image: foo bar: image: bar profiles: - test baz: image: baz depends_on: - bar profiles: - test zot: image: zot depends_on: - bar profiles: - debug In the above example: If the Compose application model is parsed with no profile enabled, it only contains the foo service. If the profile test is enabled, the model contains the services bar and baz , and service foo , which is always enabled. If the profile debug is enabled, the model contains both foo and zot services, but not bar and baz , and as such the model is invalid regarding the depends_on constraint of zot . If the profiles debug and test are enabled, the model contains all services; foo , bar , baz and zot . If Compose is executed with bar as the explicit service to run, bar and the test profile are active even if test profile is not enabled. If Compose is executed with baz as the explicit service to run, the service baz and the profile test are active and bar is pulled in by the depends_on constraint. If Compose is executed with zot as the explicit service to run, again the model is invalid regarding the depends_on constraint of zot , since zot and bar have no common profiles listed. If Compose is executed with zot as the explicit service to run and profile test is enabled, profile debug is automatically enabled and service bar is pulled in as a dependency starting both services zot and bar . See how you can use profiles in Docker Compose .","title":"Illustrative example"},{"location":"embedding/","text":"Embedding Embedding is a way to represent categorical variables in a way that can be used by machine learning algorithms. It is a way to represent a categorical variable as a continuous vector of numbers. This is done by creating a matrix of weights that is learned during the training process. The matrix is initialized randomly and updated during training. The embedding matrix is a dense representation of the categorical variable. Leaderboard https://huggingface.co/spaces/mteb/leaderboard","title":"embedding"},{"location":"embedding/#embedding","text":"Embedding is a way to represent categorical variables in a way that can be used by machine learning algorithms. It is a way to represent a categorical variable as a continuous vector of numbers. This is done by creating a matrix of weights that is learned during the training process. The matrix is initialized randomly and updated during training. The embedding matrix is a dense representation of the categorical variable.","title":"Embedding"},{"location":"embedding/#leaderboard","text":"https://huggingface.co/spaces/mteb/leaderboard","title":"Leaderboard"},{"location":"framework/","text":"Framework Pytorch vs Tensorflow The image range is different for each framework. In PyTorch, the image range is 0-1 while TensorFlow uses a range from 0 to 255. To use TensorFlow, we have to adapt the image range. To TF def dataset_to_tf( dataset, cols_to_retain, collate_fn, collate_fn_args, columns_to_np_types, output_signature, shuffle, batch_size, drop_remainder, ): \"\"\"Create a tf.data.Dataset from the underlying Dataset. This is a single-process method - the multiprocess equivalent is multiprocess_dataset_to_tf. Args: dataset (`Dataset`): Dataset to wrap with tf.data.Dataset. cols_to_retain (`List[str]`): Dataset column(s) to load in the tf.data.Dataset. It is acceptable to include column names that are created by the `collate_fn` and that do not exist in the original dataset. collate_fn(`Callable`): A function or callable object (such as a `DataCollator`) that will collate lists of samples into a batch. collate_fn_args (`Dict`): A `dict` of keyword arguments to be passed to the `collate_fn`. Can be empty. columns_to_np_types (`Dict[str, np.dtype]`): A `dict` mapping column names to numpy dtypes. output_signature (`Dict[str, tf.TensorSpec]`): A `dict` mapping column names to `tf.TensorSpec` objects. shuffle(`bool`): Shuffle the dataset order when loading. Recommended True for training, False for validation/evaluation. batch_size (`int`): Size of batches to load from the dataset. drop_remainder(`bool`, default `None`): Drop the last incomplete batch when loading. If not provided, defaults to the same setting as shuffle. Returns: `tf.data.Dataset` \"\"\" if config.TF_AVAILABLE: import tensorflow as tf else: raise ImportError(\"Called a Tensorflow-specific function but Tensorflow is not installed.\") getter_fn = partial( np_get_batch, dataset=dataset, cols_to_retain=cols_to_retain, collate_fn=collate_fn, collate_fn_args=collate_fn_args, columns_to_np_types=columns_to_np_types, return_dict=False, # TF expects numpy_function to return a list and will not accept a dict ) @tf.function(input_signature=[tf.TensorSpec(None, tf.int64)]) def fetch_function(indices): output = tf.numpy_function( getter_fn, inp=[indices], # This works because dictionaries always output in the same order Tout=[tf.dtypes.as_dtype(dtype) for dtype in columns_to_np_types.values()], ) return {key: output[i] for i, key in enumerate(columns_to_np_types.keys())} tf_dataset = tf.data.Dataset.from_tensor_slices(np.arange(len(dataset), dtype=np.int64)) if shuffle: tf_dataset = tf_dataset.shuffle(len(dataset)) tf_dataset = tf_dataset.batch(batch_size, drop_remainder=drop_remainder).map(fetch_function) def ensure_shapes(input_dict): return {key: tf.ensure_shape(val, output_signature[key].shape) for key, val in input_dict.items()} return tf_dataset.map(ensure_shapes) pytorch dataset to tf dataset import tensorflow as tf import torch # Assume that we have a PyTorch Dataset object called 'dataset' def pytorch_dataset_to_tensorflow_dataset(dataset): def generator(): for data in dataset: # Convert data from PyTorch tensors to TensorFlow tensors data = [tf.convert_to_tensor(x) for x in data] yield data # Create a TensorFlow Dataset from the generator dataset = tf.data.Dataset.from_generator(generator, output_types=data[0].dtype, output_shapes=data[0].shape) return dataset # Create a TensorFlow Dataset from the PyTorch Dataset dataset = pytorch_dataset_to_tensorflow_dataset(dataset) # Create a TensorFlow DataLoader from the TensorFlow Dataset dataloader = tf.data.DataLoader(dataset, batch_size=32, num_parallel_calls=tf.data.AUTOTUNE) image = tf.io.read_file(filename=filepath) image = tf.image.decode_jpeg(image, channels=3) #or decode_png The opposite of unsqueeze and squeeze is expand_dims : img = tf.expand_dims(img,axis=0) yield the desired/necessary transformations. As for the photos, I am quite sure that you missed a /255.0 in case of PyTorch or added a 255.0 division in case of TensorFlow. In fact, when digging deep into the Keras backend, you can see that when you call your preprocessing function, it will call this function here: def _preprocess_numpy_input(x, data_format, mode): \"\"\"Preprocesses a Numpy array encoding a batch of images. Arguments: x: Input array, 3D or 4D. data_format: Data format of the image array. mode: One of \"caffe\", \"tf\" or \"torch\". - caffe: will convert the images from RGB to BGR, then will zero-center each color channel with respect to the ImageNet dataset, without scaling. - tf: will scale pixels between -1 and 1, sample-wise. - torch: will scale pixels between 0 and 1 and then will normalize each channel with respect to the ImageNet dataset. Returns: Preprocessed Numpy array. \"\"\" if not issubclass(x.dtype.type, np.floating): x = x.astype(backend.floatx(), copy=False) if mode == 'tf': x /= 127.5 x -= 1. return x elif mode == 'torch': x /= 255. mean = [0.485, 0.456, 0.406] std = [0.229, 0.224, 0.225] else: if data_format == 'channels_first': # 'RGB'->'BGR' if x.ndim == 3: x = x[::-1, ...] else: x = x[:, ::-1, ...] else: # 'RGB'->'BGR' x = x[..., ::-1] mean = [103.939, 116.779, 123.68] std = None # Zero-center by mean pixel if data_format == 'channels_first': if x.ndim == 3: x[0, :, :] -= mean[0] x[1, :, :] -= mean[1] x[2, :, :] -= mean[2] if std is not None: x[0, :, :] /= std[0] x[1, :, :] /= std[1] x[2, :, :] /= std[2] else: x[:, 0, :, :] -= mean[0] x[:, 1, :, :] -= mean[1] x[:, 2, :, :] -= mean[2] if std is not None: x[:, 0, :, :] /= std[0] x[:, 1, :, :] /= std[1] x[:, 2, :, :] /= std[2] else: x[..., 0] -= mean[0] x[..., 1] -= mean[1] x[..., 2] -= mean[2] if std is not None: x[..., 0] /= std[0] x[..., 1] /= std[1] x[..., 2] /= std[2] return x mean and std mean = 0.0 std = 0.0 for images, _ in dl: batch_samples = images.size(0) # batch size (the last batch can have smaller size!) images = images.view(batch_samples, images.size(1), -1) mean += images.mean(2).sum(0) std += images.std(2).sum(0) mean /= len(dl.dataset) std /= len(dl.dataset) DataGenerator(keras.utils.Sequence): import numpy as np import keras class DataGenerator(keras.utils.Sequence): 'Generates data for Keras' def __init__(self, list_IDs, labels, batch_size=32, dim=(32,32,32), n_channels=1, n_classes=10, shuffle=True): 'Initialization' self.dim = dim self.batch_size = batch_size self.labels = labels self.list_IDs = list_IDs self.n_channels = n_channels self.n_classes = n_classes self.shuffle = shuffle self.on_epoch_end() def __len__(self): 'Denotes the number of batches per epoch' return int(np.floor(len(self.list_IDs) / self.batch_size)) def __getitem__(self, index): 'Generate one batch of data' # Generate indexes of the batch indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size] # Find list of IDs list_IDs_temp = [self.list_IDs[k] for k in indexes] # Generate data X, y = self.__data_generation(list_IDs_temp) return X, y def on_epoch_end(self): 'Updates indexes after each epoch' self.indexes = np.arange(len(self.list_IDs)) if self.shuffle == True: np.random.shuffle(self.indexes) def __data_generation(self, list_IDs_temp): 'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels) # Initialization X = np.empty((self.batch_size, *self.dim, self.n_channels)) y = np.empty((self.batch_size), dtype=int) # Generate data for i, ID in enumerate(list_IDs_temp): # Store sample X[i,] = np.load('data/' + ID + '.npy') # Store class y[i] = self.labels[ID] return X, keras.utils.to_categorical(y, num_classes=self.n_classes) class ImageDataGenerator(tf.keras.preprocessing.image.ImageDataGenerator): def __init__(self): super().__init__( rescale=1.0 / 255.0, rotation_range=10, width_shift_range=0.2, height_shift_range=0.2, zoom_range=[0.95, 1.05], shear_range=0.1, fill_mode=\"wrap\", horizontal_flip=True, vertical_flip=True, ) class Generator(object): def __init__(self, batch_size, name_x, name_y): data_f = None # h5py.File(open_directory, \"r\") self.x = data_f[name_x] self.y = data_f[name_y] if len(self.x.shape) == 4: self.shape_x = (None, self.x.shape[1], self.x.shape[2], self.x.shape[3]) if len(self.x.shape) == 3: self.shape_x = (None, self.x.shape[1], self.x.shape[2]) if len(self.y.shape) == 4: self.shape_y = (None, self.y.shape[1], self.y.shape[2], self.y.shape[3]) if len(self.y.shape) == 3: self.shape_y = (None, self.y.shape[1], self.y.shape[2]) self.num_samples = self.x.shape[0] self.batch_size = batch_size self.epoch_size = self.num_samples // self.batch_size + 1 * ( self.num_samples % self.batch_size != 0 ) self.pointer = 0 self.sample_nums = np.arange(0, self.num_samples) np.random.shuffle(self.sample_nums) def data_generator(self): for batch_num in range(self.epoch_size): x = [] y = [] for elem_num in range(self.batch_size): sample_num = self.sample_nums[self.pointer] x += [self.x[sample_num]] y += [self.y[sample_num]] self.pointer += 1 if self.pointer == self.num_samples: self.pointer = 0 np.random.shuffle(self.sample_nums) break x = np.array(x, dtype=np.float32) y = np.array(y, dtype=np.float32) yield x, y def get_dataset(self): dataset = tf.data.Dataset.from_generator( self.data_generator, output_signature=( tf.TensorSpec(shape=(), dtype=tf.int32), tf.RaggedTensorSpec(shape=(2, None), dtype=tf.int32), ), ) dataset = dataset.prefetch(1) return dataset def _load_image(self, image_path): image = cv2.imread(image_path) # BGR # image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # image = tf.io.read_file(image_path) # image = tf.io.decode_image( # image, # channels=self.num_channels, # dtype=tf.dtypes.uint8, # expand_animations=False, # ) # image = tf.image.resize( # image, # self.dim, # method=tf.image.ResizeMethod.BILINEAR, # preserve_aspect_ratio=True, # antialias=False, # name=None, # ) # if not issubclass(image.dtype.type, np.floating): image = image.astype(np.float32) # image = image.astype(tf.keras.backend.floatx(), copy=False) image = self.apply_image_transforms(image) # 'RGB'->'BGR' # image = image[..., ::-1] # image = tf.image.convert_image_dtype(image, dtype=tf.uint8, saturate=False) # image = tf.cast(image, tf.float32) # / 127.5 # image -= 1.0 mean = [103.939, 116.779, 123.68] # mean_tensor = tf.keras.backend.constant(-np.array(mean)) # if tf.keras.backend.dtype(image) != tf.keras.backend.dtype(mean_tensor): # image = tf.keras.backend.bias_add( # image, # tf.keras.backend.cast(mean_tensor, tf.keras.backend.dtype(image)), # data_format=\"channels_last\", # ) # else: # image = tf.keras.backend.bias_add(image, mean_tensor, \"channels_last\") # image[0, :, :] -= mean[0] # image[1, :, :] -= mean[1] # image[2, :, :] -= mean[2] image[..., 0] -= mean[0] image[..., 1] -= mean[1] image[..., 2] -= mean[2] # image = tf.keras.applications.vgg16.preprocess_input(image) \"\"\" Preprocessed numpy.array or a tf.Tensor with type float32. The images are converted from RGB to BGR, then each color channel is zero-centered with respect to the ImageNet dataset, without scaling. \"\"\" return image Unfreeze specific layers Here is one way to unfreeze specific layers. We pick the same model and some layers (e.g. block14_sepconv2 ). The purpose is to unfreeze these layers and make the rest of the layers freeze . from tensorflow import keras base_model = keras.applications.Xception( weights='imagenet', input_shape=(150,150,3), include_top=False ) # free all layer except the desired layers # which is in [ ... ] for layer in base_model.layers: if layer.name not in ['block14_sepconv2', 'block13_sepconv1']: layer.trainable = False if layer.trainable: print(layer.name) block14_sepconv2 block13_sepconv1 Compute the trainable and non-trainable variables. import tensorflow.keras.backend as K import numpy as np trainable_count = np.sum([K.count_params(w) \\ for w in base_model.trainable_weights]) non_trainable_count = np.sum([K.count_params(w) \\ for w in base_model.non_trainable_weights]) print('Total params: {:,}'.format(trainable_count + non_trainable_count)) print('Trainable params: {:,}'.format(trainable_count)) print('Non-trainable params: {:,}'.format(non_trainable_count)) Total params: 20,861,480 Trainable params: 3,696,088 Non-trainable params: 17,165,392 tensorflow-macos Releases tensorflow-macos tensorflow-metal macOS version Features v2.5 0.1.2 12.0+ Pluggable device v2.6 0.2.0 12.0+ Variable sequences for RNN layers v2.7 0.3.0 12.0+ Custom op support v2.8 0.4.0 12.0+ RNN performance improvements v2.9 0.5.0 12.1+ Distributed training","title":"Framework"},{"location":"framework/#framework","text":"","title":"Framework"},{"location":"framework/#pytorch-vs-tensorflow","text":"The image range is different for each framework. In PyTorch, the image range is 0-1 while TensorFlow uses a range from 0 to 255. To use TensorFlow, we have to adapt the image range.","title":"Pytorch vs Tensorflow"},{"location":"framework/#to-tf","text":"def dataset_to_tf( dataset, cols_to_retain, collate_fn, collate_fn_args, columns_to_np_types, output_signature, shuffle, batch_size, drop_remainder, ): \"\"\"Create a tf.data.Dataset from the underlying Dataset. This is a single-process method - the multiprocess equivalent is multiprocess_dataset_to_tf. Args: dataset (`Dataset`): Dataset to wrap with tf.data.Dataset. cols_to_retain (`List[str]`): Dataset column(s) to load in the tf.data.Dataset. It is acceptable to include column names that are created by the `collate_fn` and that do not exist in the original dataset. collate_fn(`Callable`): A function or callable object (such as a `DataCollator`) that will collate lists of samples into a batch. collate_fn_args (`Dict`): A `dict` of keyword arguments to be passed to the `collate_fn`. Can be empty. columns_to_np_types (`Dict[str, np.dtype]`): A `dict` mapping column names to numpy dtypes. output_signature (`Dict[str, tf.TensorSpec]`): A `dict` mapping column names to `tf.TensorSpec` objects. shuffle(`bool`): Shuffle the dataset order when loading. Recommended True for training, False for validation/evaluation. batch_size (`int`): Size of batches to load from the dataset. drop_remainder(`bool`, default `None`): Drop the last incomplete batch when loading. If not provided, defaults to the same setting as shuffle. Returns: `tf.data.Dataset` \"\"\" if config.TF_AVAILABLE: import tensorflow as tf else: raise ImportError(\"Called a Tensorflow-specific function but Tensorflow is not installed.\") getter_fn = partial( np_get_batch, dataset=dataset, cols_to_retain=cols_to_retain, collate_fn=collate_fn, collate_fn_args=collate_fn_args, columns_to_np_types=columns_to_np_types, return_dict=False, # TF expects numpy_function to return a list and will not accept a dict ) @tf.function(input_signature=[tf.TensorSpec(None, tf.int64)]) def fetch_function(indices): output = tf.numpy_function( getter_fn, inp=[indices], # This works because dictionaries always output in the same order Tout=[tf.dtypes.as_dtype(dtype) for dtype in columns_to_np_types.values()], ) return {key: output[i] for i, key in enumerate(columns_to_np_types.keys())} tf_dataset = tf.data.Dataset.from_tensor_slices(np.arange(len(dataset), dtype=np.int64)) if shuffle: tf_dataset = tf_dataset.shuffle(len(dataset)) tf_dataset = tf_dataset.batch(batch_size, drop_remainder=drop_remainder).map(fetch_function) def ensure_shapes(input_dict): return {key: tf.ensure_shape(val, output_signature[key].shape) for key, val in input_dict.items()} return tf_dataset.map(ensure_shapes)","title":"To TF"},{"location":"framework/#pytorch-dataset-to-tf-dataset","text":"import tensorflow as tf import torch # Assume that we have a PyTorch Dataset object called 'dataset' def pytorch_dataset_to_tensorflow_dataset(dataset): def generator(): for data in dataset: # Convert data from PyTorch tensors to TensorFlow tensors data = [tf.convert_to_tensor(x) for x in data] yield data # Create a TensorFlow Dataset from the generator dataset = tf.data.Dataset.from_generator(generator, output_types=data[0].dtype, output_shapes=data[0].shape) return dataset # Create a TensorFlow Dataset from the PyTorch Dataset dataset = pytorch_dataset_to_tensorflow_dataset(dataset) # Create a TensorFlow DataLoader from the TensorFlow Dataset dataloader = tf.data.DataLoader(dataset, batch_size=32, num_parallel_calls=tf.data.AUTOTUNE) image = tf.io.read_file(filename=filepath) image = tf.image.decode_jpeg(image, channels=3) #or decode_png The opposite of unsqueeze and squeeze is expand_dims : img = tf.expand_dims(img,axis=0) yield the desired/necessary transformations. As for the photos, I am quite sure that you missed a /255.0 in case of PyTorch or added a 255.0 division in case of TensorFlow. In fact, when digging deep into the Keras backend, you can see that when you call your preprocessing function, it will call this function here: def _preprocess_numpy_input(x, data_format, mode): \"\"\"Preprocesses a Numpy array encoding a batch of images. Arguments: x: Input array, 3D or 4D. data_format: Data format of the image array. mode: One of \"caffe\", \"tf\" or \"torch\". - caffe: will convert the images from RGB to BGR, then will zero-center each color channel with respect to the ImageNet dataset, without scaling. - tf: will scale pixels between -1 and 1, sample-wise. - torch: will scale pixels between 0 and 1 and then will normalize each channel with respect to the ImageNet dataset. Returns: Preprocessed Numpy array. \"\"\" if not issubclass(x.dtype.type, np.floating): x = x.astype(backend.floatx(), copy=False) if mode == 'tf': x /= 127.5 x -= 1. return x elif mode == 'torch': x /= 255. mean = [0.485, 0.456, 0.406] std = [0.229, 0.224, 0.225] else: if data_format == 'channels_first': # 'RGB'->'BGR' if x.ndim == 3: x = x[::-1, ...] else: x = x[:, ::-1, ...] else: # 'RGB'->'BGR' x = x[..., ::-1] mean = [103.939, 116.779, 123.68] std = None # Zero-center by mean pixel if data_format == 'channels_first': if x.ndim == 3: x[0, :, :] -= mean[0] x[1, :, :] -= mean[1] x[2, :, :] -= mean[2] if std is not None: x[0, :, :] /= std[0] x[1, :, :] /= std[1] x[2, :, :] /= std[2] else: x[:, 0, :, :] -= mean[0] x[:, 1, :, :] -= mean[1] x[:, 2, :, :] -= mean[2] if std is not None: x[:, 0, :, :] /= std[0] x[:, 1, :, :] /= std[1] x[:, 2, :, :] /= std[2] else: x[..., 0] -= mean[0] x[..., 1] -= mean[1] x[..., 2] -= mean[2] if std is not None: x[..., 0] /= std[0] x[..., 1] /= std[1] x[..., 2] /= std[2] return x","title":"pytorch dataset to tf dataset"},{"location":"framework/#mean-and-std","text":"mean = 0.0 std = 0.0 for images, _ in dl: batch_samples = images.size(0) # batch size (the last batch can have smaller size!) images = images.view(batch_samples, images.size(1), -1) mean += images.mean(2).sum(0) std += images.std(2).sum(0) mean /= len(dl.dataset) std /= len(dl.dataset)","title":"mean and std"},{"location":"framework/#datageneratorkerasutilssequence","text":"import numpy as np import keras class DataGenerator(keras.utils.Sequence): 'Generates data for Keras' def __init__(self, list_IDs, labels, batch_size=32, dim=(32,32,32), n_channels=1, n_classes=10, shuffle=True): 'Initialization' self.dim = dim self.batch_size = batch_size self.labels = labels self.list_IDs = list_IDs self.n_channels = n_channels self.n_classes = n_classes self.shuffle = shuffle self.on_epoch_end() def __len__(self): 'Denotes the number of batches per epoch' return int(np.floor(len(self.list_IDs) / self.batch_size)) def __getitem__(self, index): 'Generate one batch of data' # Generate indexes of the batch indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size] # Find list of IDs list_IDs_temp = [self.list_IDs[k] for k in indexes] # Generate data X, y = self.__data_generation(list_IDs_temp) return X, y def on_epoch_end(self): 'Updates indexes after each epoch' self.indexes = np.arange(len(self.list_IDs)) if self.shuffle == True: np.random.shuffle(self.indexes) def __data_generation(self, list_IDs_temp): 'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels) # Initialization X = np.empty((self.batch_size, *self.dim, self.n_channels)) y = np.empty((self.batch_size), dtype=int) # Generate data for i, ID in enumerate(list_IDs_temp): # Store sample X[i,] = np.load('data/' + ID + '.npy') # Store class y[i] = self.labels[ID] return X, keras.utils.to_categorical(y, num_classes=self.n_classes) class ImageDataGenerator(tf.keras.preprocessing.image.ImageDataGenerator): def __init__(self): super().__init__( rescale=1.0 / 255.0, rotation_range=10, width_shift_range=0.2, height_shift_range=0.2, zoom_range=[0.95, 1.05], shear_range=0.1, fill_mode=\"wrap\", horizontal_flip=True, vertical_flip=True, ) class Generator(object): def __init__(self, batch_size, name_x, name_y): data_f = None # h5py.File(open_directory, \"r\") self.x = data_f[name_x] self.y = data_f[name_y] if len(self.x.shape) == 4: self.shape_x = (None, self.x.shape[1], self.x.shape[2], self.x.shape[3]) if len(self.x.shape) == 3: self.shape_x = (None, self.x.shape[1], self.x.shape[2]) if len(self.y.shape) == 4: self.shape_y = (None, self.y.shape[1], self.y.shape[2], self.y.shape[3]) if len(self.y.shape) == 3: self.shape_y = (None, self.y.shape[1], self.y.shape[2]) self.num_samples = self.x.shape[0] self.batch_size = batch_size self.epoch_size = self.num_samples // self.batch_size + 1 * ( self.num_samples % self.batch_size != 0 ) self.pointer = 0 self.sample_nums = np.arange(0, self.num_samples) np.random.shuffle(self.sample_nums) def data_generator(self): for batch_num in range(self.epoch_size): x = [] y = [] for elem_num in range(self.batch_size): sample_num = self.sample_nums[self.pointer] x += [self.x[sample_num]] y += [self.y[sample_num]] self.pointer += 1 if self.pointer == self.num_samples: self.pointer = 0 np.random.shuffle(self.sample_nums) break x = np.array(x, dtype=np.float32) y = np.array(y, dtype=np.float32) yield x, y def get_dataset(self): dataset = tf.data.Dataset.from_generator( self.data_generator, output_signature=( tf.TensorSpec(shape=(), dtype=tf.int32), tf.RaggedTensorSpec(shape=(2, None), dtype=tf.int32), ), ) dataset = dataset.prefetch(1) return dataset def _load_image(self, image_path): image = cv2.imread(image_path) # BGR # image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # image = tf.io.read_file(image_path) # image = tf.io.decode_image( # image, # channels=self.num_channels, # dtype=tf.dtypes.uint8, # expand_animations=False, # ) # image = tf.image.resize( # image, # self.dim, # method=tf.image.ResizeMethod.BILINEAR, # preserve_aspect_ratio=True, # antialias=False, # name=None, # ) # if not issubclass(image.dtype.type, np.floating): image = image.astype(np.float32) # image = image.astype(tf.keras.backend.floatx(), copy=False) image = self.apply_image_transforms(image) # 'RGB'->'BGR' # image = image[..., ::-1] # image = tf.image.convert_image_dtype(image, dtype=tf.uint8, saturate=False) # image = tf.cast(image, tf.float32) # / 127.5 # image -= 1.0 mean = [103.939, 116.779, 123.68] # mean_tensor = tf.keras.backend.constant(-np.array(mean)) # if tf.keras.backend.dtype(image) != tf.keras.backend.dtype(mean_tensor): # image = tf.keras.backend.bias_add( # image, # tf.keras.backend.cast(mean_tensor, tf.keras.backend.dtype(image)), # data_format=\"channels_last\", # ) # else: # image = tf.keras.backend.bias_add(image, mean_tensor, \"channels_last\") # image[0, :, :] -= mean[0] # image[1, :, :] -= mean[1] # image[2, :, :] -= mean[2] image[..., 0] -= mean[0] image[..., 1] -= mean[1] image[..., 2] -= mean[2] # image = tf.keras.applications.vgg16.preprocess_input(image) \"\"\" Preprocessed numpy.array or a tf.Tensor with type float32. The images are converted from RGB to BGR, then each color channel is zero-centered with respect to the ImageNet dataset, without scaling. \"\"\" return image","title":"DataGenerator(keras.utils.Sequence):"},{"location":"framework/#unfreeze-specific-layers","text":"Here is one way to unfreeze specific layers. We pick the same model and some layers (e.g. block14_sepconv2 ). The purpose is to unfreeze these layers and make the rest of the layers freeze . from tensorflow import keras base_model = keras.applications.Xception( weights='imagenet', input_shape=(150,150,3), include_top=False ) # free all layer except the desired layers # which is in [ ... ] for layer in base_model.layers: if layer.name not in ['block14_sepconv2', 'block13_sepconv1']: layer.trainable = False if layer.trainable: print(layer.name) block14_sepconv2 block13_sepconv1","title":"Unfreeze specific layers"},{"location":"framework/#compute-the-trainable-and-non-trainable-variables","text":"import tensorflow.keras.backend as K import numpy as np trainable_count = np.sum([K.count_params(w) \\ for w in base_model.trainable_weights]) non_trainable_count = np.sum([K.count_params(w) \\ for w in base_model.non_trainable_weights]) print('Total params: {:,}'.format(trainable_count + non_trainable_count)) print('Trainable params: {:,}'.format(trainable_count)) print('Non-trainable params: {:,}'.format(non_trainable_count)) Total params: 20,861,480 Trainable params: 3,696,088 Non-trainable params: 17,165,392","title":"Compute the trainable and non-trainable variables."},{"location":"framework/#tensorflow-macos-releases","text":"tensorflow-macos tensorflow-metal macOS version Features v2.5 0.1.2 12.0+ Pluggable device v2.6 0.2.0 12.0+ Variable sequences for RNN layers v2.7 0.3.0 12.0+ Custom op support v2.8 0.4.0 12.0+ RNN performance improvements v2.9 0.5.0 12.1+ Distributed training","title":"tensorflow-macos Releases"},{"location":"git/","text":"GIT README.md \ud83d\udc4b Hi, I\u2019m @furyhawk \ud83d\udc40 I\u2019m interested in AI \ud83c\udf31 I\u2019m currently learning AI \ud83d\udc9e\ufe0f I\u2019m looking to collaborate on AI \ud83d\udceb How to reach me ... https://github.com/furyhawk git config --global user.name \"furyhawk\" git config --global user.email furyx@hotmail.com git config pull.rebase true commit msg : \u2502 \u2502 \u2502 \u2514\u2500\u2af8 Summary in present tense. Not capitalized. No period at the end. \u2502 \u2514\u2500\u2af8 Commit Type: build|cicd|docs|feat|fix|node|refactor|test Fetch/pull all branches git branch -r | grep -v '\\->' | sed \"s,\\x1B\\[[0-9;]*[a-zA-Z],,g\" | while read remote; do git branch --track \"${remote#origin/}\" \"$remote\"; done git fetch --all git pull --all How can I enable github notifications? install and authenticate with the github cli: pacman -S github-cli gh auth login ubuntu: type -p curl >/dev/null || (sudo apt update && sudo apt install curl -y) curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg \\ && sudo chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg \\ && echo \"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main\" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null \\ && sudo apt update \\ && sudo apt install gh -y reset to last working commit: git reset --hard git push --force working with submodules: git submodule update --init --recursive There are four steps involved when you delete a submodule. # 1. deinit the submodule git submodule deinit # 2. Remove the submodule directory from Git git rm # 3. Remove the submodule directory from .git/modules/ rm -rf .git/modules/ # 4. commit and push the changes # add submodule and define the master branch as the one you want to track git submodule add -b master [URL to Git repo] git submodule init","title":"git"},{"location":"git/#git","text":"","title":"GIT"},{"location":"git/#readmemd","text":"\ud83d\udc4b Hi, I\u2019m @furyhawk \ud83d\udc40 I\u2019m interested in AI \ud83c\udf31 I\u2019m currently learning AI \ud83d\udc9e\ufe0f I\u2019m looking to collaborate on AI \ud83d\udceb How to reach me ... https://github.com/furyhawk git config --global user.name \"furyhawk\" git config --global user.email furyx@hotmail.com git config pull.rebase true","title":"README.md"},{"location":"git/#commit-msg","text":": \u2502 \u2502 \u2502 \u2514\u2500\u2af8 Summary in present tense. Not capitalized. No period at the end. \u2502 \u2514\u2500\u2af8 Commit Type: build|cicd|docs|feat|fix|node|refactor|test","title":"commit msg"},{"location":"git/#fetchpull-all-branches","text":"git branch -r | grep -v '\\->' | sed \"s,\\x1B\\[[0-9;]*[a-zA-Z],,g\" | while read remote; do git branch --track \"${remote#origin/}\" \"$remote\"; done git fetch --all git pull --all","title":"Fetch/pull all branches"},{"location":"git/#how-can-i-enable-github-notifications","text":"install and authenticate with the github cli: pacman -S github-cli gh auth login ubuntu: type -p curl >/dev/null || (sudo apt update && sudo apt install curl -y) curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg \\ && sudo chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg \\ && echo \"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main\" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null \\ && sudo apt update \\ && sudo apt install gh -y reset to last working commit: git reset --hard git push --force working with submodules: git submodule update --init --recursive There are four steps involved when you delete a submodule. # 1. deinit the submodule git submodule deinit # 2. Remove the submodule directory from Git git rm # 3. Remove the submodule directory from .git/modules/ rm -rf .git/modules/ # 4. commit and push the changes # add submodule and define the master branch as the one you want to track git submodule add -b master [URL to Git repo] git submodule init","title":"How can I enable github notifications?"},{"location":"gpt/","text":"gpt https://github.com/furyhawk/nanoGPT train python train.py --dataset=shakespeare --n_layer=4 --n_head=4 --n_embd=64 --device=mps --compile=False --eval_iters=1 --block_size=64 --batch_size=16 logs iter 432061: loss 3.3774, time 102.72ms number of parameters: 3.42M No meta.pkl found, assuming GPT-2 encodings... The state of heaven with him yet have set The enam received in his power, It will be such,morrow; if he be By common enemies, so may not do, you As if you mark the truth. FLORIZEL: Why is my part; Not here my life, we'll do't, where it is The lark that the Earl of Warwick is slain: On bold, my rest, my liege, in whose good time I'll drown him worthy me to my Earl towards, Is Come to draw to scARD a fearful day. 3 KING HENRY VI KING LEWIS XI: What! canst thou live in poor soul, To do him banish'd at the same our point And the freshinks I come, 'tis aught to have but I fear you joys: all that I cannot On thee would of my heart, my poor soul, To be so bold a knave in this time; And thou by, she's children's blood, only, it is To see my life. MARIANA: A thousand part, indeed, sir, a time As you will wonder not: come, my liege, I well; And he is too a gentleman. See that with the voice that at once, you must Flried the senators of the any it good to us being son: but, I cannot be done; but we can tell him; for him I'll draw your words together. What you will know how you? COMINIUS: They are, uncle. CORIOLANUS: Therefore, my gracious friend, I am the farther un ches: What, poor! I beseech you, let it be a cousin. VOLUMNIA: Tush, the gods! MENENIUS: I am too! Nay begin to bid my lady play well: I am no pity that willy heir is right; But, by his leave I have or why it is. CORIOLANUS: Not? let me see, you are strange? They have made no more than all. Hath my wife's lord, upon pain of death. Third Citizen: Why, I shall have had some heard of you hear. QUEEN MARGARET: And so we all, I thank thee: 'tis more; Thou art the letter, and thou hast not so: wouldst be done, hark you, I know, my soul here have t something So to say, but I pray, and no more. ISABELLA: O, if it were full of such deep Norfolk, love, I would have more than his gracious fault Than Angelo first to you at great comfort. ANGELO: Yea, and make you he crook 'pt TrleKE: I'll make. Provost: What, my general? DUKE VINCENTIO: 'Tis like of his most, And leave me to the will for my poor side. ANGELO: Hath the king'ster heaven with the majesty of tears, With a kind: I have, my lord, must die. MARCIUS: Tell me hate them not, honesty are there To entreat of my wife, and my son to my order, And set the mother all my father's consent To make the must fly: so go: From whom that I would not, I do not fear, As well I will have me, you are vile toward. QUEEN MARGARET: But, by your vouchius, I fear not what I am: For since we came, your father's, at the least? Second Servant: Tis more than this of you can but have done My love or mercy to the queen of heaven: But I have forgot it, more, goodenio. CAPULET: Ay, as the matter I told you speak, my lord. BALTHASAR: But to the case of the Tower. ROMEO: Never; live, my lord, my soul is as well: I hope, by this which is a villain, When that thy brothers Romeo is, one sun With thy mother's wife, that I may return To know me what a dream hath as well As she run thought to be your mother, I I may not need to my sweet, CAPULET: Ay, what art thou? ROMEO: A queen, nurse? JULIET: Good lords, If thou dost know ye but. JULIET: Say, that Richmond's soul, take me in all. ROMEO: O, Romeo, I will not weep a while. JULIET: For the defry of grief makes them fair: But since unsapblful villain's wit of heaven It was a friend for ever-day, When I was lawful oath and not so, I am born to do it would it might be, I am revenged on-hearted my leaves dead king's oath, For the wantless blood whose heads must enter: 'Tis the Lord Northumberland that King Edward was himself, The Duke of Warwick, and his place in France, And thou shalt come to my will-day shall not stay. HENRY BOLINGBROKE: Why, howof, I beseech you talk of, Here must I play my heart with a cruelay; That doth not fall dead, I know the matter. DUKE VINCENTIO: You had even like'd, fie me, 'tis, I am resolved: Nay, unadBe now, I have heard her speak more than you I speak at peace. ANGELO: Well, I do beseech you, fair sir. Your eyes will not come upon me: he is most man, it is not so: so most much little most wrong, and may go to that by: which you have therein your hate, we may see themselves. LADY CAPULET: We are in prison, one word; ask him; For some you win me, upon his best: You know no better for the mind, there's no matter stands on him. DUCHESS OF YORK: How wouldst thou? Messenger: My gracious lord, I like youThis. DUKE OF YORK: Is it not dead, cousin. DUCHESS OF YORK: Then 'tis shame forAnd all that thou wast? KING R DUKE VINCENTIO: Romeo. Provost: It do trust me. ANGELO: Our lord, I cannot speak the good word: Since I havewas not it, I do'll therefore know Will a divineade me speak. ISABELLA: O, let me hear what you did deliver, By 'twere pity to him. This friends, be his head it, with one that's no matter, look to me as this lure of the prince a curse; more stands on't; No, sir; not a very weak of mine, If not that. Now, andso death's like death! I see, and hear Montague, my Montague. Who knows thou nothing; be not gentle, I may not, yet they shall be so, then; And had he come'd withAnd the oracle: I'll have your heart: that I shall follow that send him; that, we will warrant you, andPEY? ELBOW: Nay, he shall get a man to me. ELBOW: Look, very well, he shallEL' the better royal, which I do do't But I shall find, a time will be from his spirit. I prithee, take one: give me not your comfortier: I will not be come to-morrow; let's entreaty? ROMEO: I beseech your grace, sir,-- BISHOP OF CARke, my dTwere 't. BUCKINGHAM: I will, my lord: nay, You will have you go with me to please again. BUCKINGHAM: You will not hear duty, but for I will, sir, You have not the lady will with thee love But what you may! You are sure of he is to be a doth valiant; I will wot the prince: and have first to-morrow then. ANGELO: Thy business is the more and that my brother's sister. ISABELLA: It cannot be so. DUKE VINCENTIO: I say you, well; for if his present is the WARWICK: And thou wert enam'd; I'll not be mine own? KING LEWIS XI: What man is none than that at his house, That bear their down heart to Edward's-t thou him? WARWICK: Bid me, noble lord, our friends are fled, And sit to be true; and with all our sonsiss, From forth the most defend in theile departedath Be not born to make a husband as if I send it to thought that I did weep, And quench blood or not well. KING RICHARD II: O I was Clarence! What'sFL words were it: That I, as we are, that they have no friend, Even in the blood of heaven of fight, Thou p country's blood and'd. Come, and go with thee go; And, as mine I remember as any man's son As merry as I told the people and am mighty As'd an hour begin and bears our hands, But shame no other Paris and her be drawn. ROMEO: Is my long come. carry, let'st thou father In this remembrance of love: thou eat not 'em most noble swear tongue for what thou mayst, not wert between. The Earl of heaven of Warwick's love, this man, That's hearts, the true king, myMON and Sir. And unlook'd in the princess slain, Is beg of hen, and did yield in their arms At no moreOR create give from their hands, For the dead George of Clarence to his king, Who spake me his enemies shall hold it. BUCKINGHAM: My lord, my gracious lord, You had a power of wisdom cam thou out of mine? KING RICHARD II: He hath enem this, people, but I'll win me. CATESBY: Then here, my lord. KING RICHARD III: Nor I, the crown that queen is slain, To take the devil of our other linean ground, Not his our kingly curse people's-Which they that Is put to have open honesty Either to come in their arms. QUEEN MARGARET: ANTIGONUS: Hear you not? First Citizen: Come, sir; go, be it must none but you: I am ta'en and leave of you; and, then, go with me. First Servingman: Why, we will, sir. Second Servingman: I would not, no more: the provost. Third Servingman: What's the matter? Third Servingman: What's the matter? Second Servingman: No, my master, I can tell; I know how it you, thereof he m it. Clown: He's here; we should be a witness to the purpose. Third Citizen: He hath done what you are done. Third Servingman: A Romeo, sir, for them he would have found baw Rome of the world. CORIOLANUS: But I love the child, Not that thou hast other of thy life. Third Servingman: If thou she were thrice a man, And had these griefing force royal royal queen; And I am son, and I love thee myself. KING RICHARD III: But I will be poor; which he is he made? BUCKINGHAM: My gracious lord, let's see your grace be company. KING RICHARD III: Why, what a fellow should be? BUCKINGHAM: My lord, 'tis a subject, proud Which else, which, had some him, it will with heaven Till wind manage our doth made over his land. Why, proud I have ta'en the service of the people, That would not have more better that, Which will be satisfied; and, how his ignorant shouldER' He seems not? and I have had rather thy, To do my country, to make the'd right, With all the envy of this loyal, Our holy and be England and Duke of York, Not in his second soul I and have been The which never bid us, and call King Richard's lord. HENRY BOLINGBROKE: Welcome, uncle; we shall not be unto him; But so, my good son, I I do further. KING RICHARD III: Why, then, I confess thee, lord, If thou hadst never yet thou yet hadst never, But yet thou wilt wert up in thy horse. CLIFFORD: Clarence and Gloucester, I will not bethink, But that's not yet did nothing: but the gods The times of revenge! RICHARD: Nay, bring me what ancient, and this hand.' KING HENRY VI: But, to thee speak commonth very day! CLIFFORD: That thou wert so disgraced me to me. KING HENRY VI: Woth he the oath that I should tread upon our way? CLIFFORD: What may your grace in poor Henry's life and him? WARWICK: No, like a bawd, you not so; and, and for the poor one did, he is even to give his l. WARWICK: Uncle, so: I do intend to him in all, I hope. KING HENRY VI: How far I mean? if this be so bold with! The which, his queen, the queen, his love; And so, gentle king, may they do you in at night And fly him in the slander of his king, To be revenged on him that, so should you have; For, by this way be you found you to, She's a woman. ANGELO: He will not see you, sir. ESCALUS: I am aTo your request in hand, you shall in arms You must give, and yet go by, a course it is thence. ESCALUS: LARTI not mock mine own good. ANGELO: Do you hear. The duke hath forgot you for what you are? ESCALUS: BUS: for whose offence? CORIOLANUS: What to you? CORIOLANUS: O, worthy madam, And have I too most, you shall not. AUFIDIUS: Worthy sir, farewell. VOL Had not himself, nor known she! Provost: An me, both that; he did, sir, to steal-on Her mother's Romeo! O old faithful friar! JULIET: Why do I more than that I should never speak of. ROMEO: Nay, rather give me leave me that I will; For I shall find some better happy days Than lay, a man. FRIAR LAURENCE: But slain, her hath the next day a little. ROMEO: Nay, good woman; my turn in thy sake art full nights here. Who is dead and young prince. Ah, how, dost thou find me to thy breast, Thine hast he, and more, a noble fawd. Sir, thou canst not speak, I wouldst thou wert For sleep the mark of thy deadly years? Ah, keep'st, asLord men, thou hast got By so thy voice: there's no power, it is none honest. MERCUTIO: Tybalt, Romeo, whom I have the best I think how I have done. ROMEO: Thy life art thou, that thou art too fair! BENVOLIO: Mehe, and go not in 't. MERCUTIO: Nay, I'll bring some noise for a for that. CLAUDIO: Your face? aarer is a traitor's head to pluck him to the king the house. But which you can do I have to the hand; for it will be, as I am and a block withFor a man that Claudio hath married, If be a uncle. DUKE VINCENTIO: He did know more. You are most little more, he hath done it well, our the city and do them to make me theD:' Aufidius come to me and brother; and she isome with those that hath been, 'twere pity us all. Then, soft, wot each that am aly; For now I did well know. My mother! LEONTES: What's well man? CAMILLO: KING RICHARD III: Be not so? HENRY BOLINGBROKE: So that of God he would have the king his! BUCKINGHAM: I'll make more than my gracious case: I will bear the deed. KING RICHARD III: tis you by king, I would am I bethink, To make an amissant in love That thou wouldst protest to keep post toOR'n; And thou he is not nowSo. LEONTES: Unman, it is; Nay, good my good lord, be satisfied still. LEONTES: A most business: Let me have been since I was to be mercy; Whiles the lark-f orth-she, He cannot, Marcius worthyKE: 'tis well stock no poison. The manner of the envy he spake to Marcius, That with the power is worth dead, the doth be brief, In worthy Romeo hastous tune up. KING RICHARD II: Whom was it so, my child is set on death. HENRY BOLINGBROKE: ClOR, my lord; With late that cons news, for, to that I would Trueass the great number pardon'd, from your head already're his mind i' the there, I will. DUKE VINCENTIO: You do but see your highness of your arms Would know the king's mouth with peace. MARIANA: I mean, a son, it is not a word through a tear for Claudio, and be banished; But when you bid this manner bless us, In whom we have had been wont to do, 'twas upon the business, you I shall call'd his friends to him, and then. DUKE OF Lord: He's tearsark; for long my gentleness' I let me hear; good cousin, adieu! Belike his hands I in his remedy; And I have died to face, my love to me, And I will make him leave of can say, I had rather keep their words with him to keep him Whither, to thy fortune but put off; And see that this terror may be so, So if that cannot: so that's the friend, That will be deservedly of your honour's life, Let thy rage here cut in his grave; One side do not. MENENIUS: Hath the people's great state by power, That they shall have set up a part; But, as minetis the nothing of a man, That is the deputy,tis well for fault, And breathed his honour with the fear of death, That all for that as you are, and we may live. BALTHASAR: A pretty fardman! RICHMOND: Bid him be here? and I'll make me pardon up. RIVERS: What, hast he not? BUCKINGHAM: My lord, my gracious lord. KING RICHARD III: As I remember; against thee, thou wert not so much done, That thou canst swear and thee, for thou hast no cause. RATCLIFF: I will not be not, my lord. KING RICHARD III: Ay, O my children, my eyes! BUCKINGHAM: No, mighty lord, I hope, shall understand it. KING RICHARD III: I know no? but himself he is done; The side must be seen and make me wrong. QUEEN ELIZABETH: That thou mean I sent to see thy life To have no more fit than it is in this world To look upon thy brother's make me hear, The Duke of Norfolk; if thou darest with Thy Lord Northumberland, rouse'd and thy crown, For thou shalt no life but by thy kingdom. KING EDWARD IV: But thou, in my turn, my grave jest thou, You, the newats, and have the ears, Where fruit men have done-bended, then my daughter Of happyness! O my woman! If thou didst, thou tis'st fair a piteous, which he was, And know not 't prove a thousand right while you That you are to have something: so, my kingdom, 'Tis just that I iter 506998: loss 3.5516, time 71.91ms iter 506999: loss 3.6803, time 72.24ms iter 507000: loss 3.3026, time 72.24ms iter 507001: loss 3.4185, time 71.62ms iter 507002: loss 3.6271, time 73.12ms iter 507003: loss 3.5836, time 71.73ms iter 507004: loss 3.1981, time 75.92ms iter 507005: loss 3.6141, time 71.98ms iter 507006: loss 3.2294, time 73.99ms number of parameters: 3.42M No meta.pkl found, assuming GPT-2 encodings... The state of heaven with him, and that his body is humray'd this. WARWICK: And I. DUCHESS OF YORK: My dear lord, DUCHESS OF YORK: HENRY BOLINGBROKE: As far go we as any we toOL? HORTENSIO: I think she is well: then will you have so: I would we were not been then in war's good. DUKE VINCENTIO: We shall not be still be gone and been to see. GREMIO: And your good lord, I would all and like you; and in your hope. ISABELLA: I can know the truth, I am your father's house. DUKE VINCENTIO: It is no gone, sir? LUCIO: I think I would, sir, I know say, I am a friendlike by him: it is the very much of your blood; if Would I be well show'd of you in the bosom And our new-a. Provost: MAMILLIUS: My lord, I'll warrant thee a holy heart: which's with mine, I am too good to him. LEONTES: I am none; Sir, for your son I have here in this business To the justice of your father's worth. PAULINA: It is my gr in that I were, To the most of your own: your therefore have Be consul: Come, come, go, go, be much here too much: I am no more. LEONTES: They are such a and the and yet. HERMIONE: The better, sir; I am an't that you have notity. LEONTES: LEONTES: Well, well, what! LARTIUS: I am so. PAULINA: I pray any I, sir? PAULINA: Alas, poor woman! What ha HENRYHAM: Too true Clarence was a holy bed. CAMILLO I shall have had a brother of your son, To yield your brother's brother. LordOUCESTER: CLARENCE: KING EDWARD IV: KING EDWARD IV: Why, then, think you what: go-morrow, son, That you must not do that I am just, For when you can, my lord, my brother, Henry With your brother, if he bring end to your last. And, high-ed, if you see her you do, He would not have been thus an whose speak With that most un Butice: I would, he were born, a name: The house of him I'll speak to see your wife's. DUKE VINCENTIO: 'Tis not her in, sir, his friends, which Of wrongstAName to the earth, with a little way. ISABELLA: I am so mither'd by the point of it That I have had. DUKE VINCENTIO: O, let me think; and let me see a word. Provost: I wouldISABELLA: Wilt thou not, then, not in a very kind for a very for me that I did use me from him to speak. And I beseech your majesty and your bosom And my great opinion, have a wife and father, Unto my brother father and my well. FRIAR LAURENCE: And there is my hNor talk'd be no sworn to death But that I should think 'twack the part of you In the duke's daughter. Nurse: A love a man that his blood is more a life Than this may standow make her false friends. Nay, more farewell; I not for't; Our unf shors'll ask you: come by the man-likeent of this lady is the least, But to the everatter of me dearest day, Or me but one earth, like a thing that Not so much sorrow's, as many more is true Than this in his own! as our bthe light As I do! What's in all my shame already may in openings! You, my sweet HASTINGS: Be early what would you say to our lord? YORK: It is not so. GLOUCESTER: It is. LADY ANNE: sweet my old grace, that thou follow'st! GLOUCESTER: And, for God's sake, till you are three years. LADY ANNE: What, prince? GLOUCESTER: No noble thing I should be a happy king? GLOUCESTER: Away with him! DUCHESS OF YORK: My lord, I must not need: Yere to my children's love's loss of thee, But thou, even to the king, thou thy doubt'st tongue. KING RICHARD III: What is't o'So? DUCHESS OF YORK: I, by my life, I have done, but so in once For in the virtue of our blood: without that kind Was never so far gGLOUCESTR well, And I will follow him. DUKE VINCENTIO: M King Richard, give me leave to my thou thou go Be at home of these good fairWARD. But of what, thou dost to do Edward! B vouch'd: My lords, more than you shall please our royal name. LORD ROMASET: So, to save the king; yet he's just, And come, to answer me in him my country, And you shall have you where we did say the prince? DUCHESS OF YORK: My Lord, may you do this dangerous lady, My father's heart, which your hand did thee yet be The first one of this noble Edward's king; And yet that is no more: come to thee, For I am we of my your love. KATHARINA: I have a very. KING RICHARD III: sadful lord, and his part-f ready I come To make the king a king from him there is That he's made good. Come, let you go; But there be 'twere noIC at you all; But you shall have France, for honour; if you were now have My body hath done to banish yourself, To prove a good will-dew heart, Wherein my noble prince was comfort To make a man's name and a great bed: 'Tis well aKINGBe, he's in a child I'll be by my husband; let me she: I'll have it if I use my sake. Why, how, let is in that sword this traitor! HENRY BOLINGBROKE: Richard, help! but more than his that I'll make my tongue to love thee such life As thou art to Romeo. KING EDWARD IV: This? but doth he not Edward's death's death! HEN MARGARET: It is more than no more yet than by it. BUCKINGHAM: What, is it so? GLOUCESTER: I shall be there was in my heart With gracious lord and my good lordle brought He is well: but I hear you will Give me to my duty, to the world. First Murderer: Because thou canst love, that ever, thy love, thy child, Thy man shall take a grave on thy wretcheding. Fare not thy sword, content Camillo, give Of your she on a my master and son; And here, I beseech your highness,-- FRIAR LAURENCE: eyeTYou may my kindred again! ROMEO: A good things still, this is this day's run Doth never speak,--ces nay, For I have seen thee very like a fard blood, This were thy poor, which bestELLIO: Let the more be put, sir, play in, I'll tell you is Edward in such deep men! KING EDWARD IV: By heaven, I hear no more: my brother's heart. HEN ELIZABETH: I am come to have a more worth in this. KING RICHARD III: Allak thou, God! it is not my son. QUEEN ELIZABETH: There is no more I swear than they shall grief is so. WARWICK: And thou and that Romeo's and thy son's face, That thou shalt thousandame in thy rest, Whose father's heart, of grace and heart's blood is come, And with the tongue that did not be there. BENVOLIO: T Gm would cause, I wouldsthip wish it true. ROMEO: Is thy life- vouch's own word. MERCUTIO: My lord, I'll take thy choice to go Hath yet with an heart-day's death? BENVOLIO: I dare not for the best upon thyPle I talk of thy Ver lives. Come, let's not A noble man: For she hath yet a man to fall? ROMEO: Thou art not so well:OM pity her: But he's an but one. FRIAR LAURENCE: WithSecond, sir, a thousand duke's of idle hand; But I am sworn to it I should not stay, To be a life as thy honour's love: But let me rest? JULIET: This will be, to put forth; there is no man. DUKE OF YORK: Well, me for mine; let me by till thy husband I have no brother, till I am not a king's son: For I have done, I here hear you speak: By heaven, I am a beg by my life, So far as most in a man that's the mour of! O, that I would be less! the great which o' the honour, And that I had; but then, I am a-beLEath slain, Were kill my takest! If never, run Aufidius, 'Tis thought it would you are at enmity That you did but so much is, and And I'll lay down out whose't be fullle In such most I have kill'd his have been such his, As so in the world is dead. Messenger: The man that he is our lady. KING RICHARD III: He kill, my lord; who, we could not say 'ld have so much. Come, come away. Of God's name and our fair? 'Tis oath? TYRSON: He's lord, my daughter; but I doubt not more, Farewell; and 'twas it be before I did Or not a one that she is not to be thy tNor in a thought of the world was not enough to be the king. Provost: DUKE VINCENTIO: How? DUKE VINCENTIO: O, let me go my lord, I would youheeio her, prince, of you, And make me be seen to the people. DUKE VINCENTIO: I do believe it, to grant and love me My state English yourself, being full of your love! O prince! one of you a man that I must not all be the king, which is mine and him. LUCIO: But, madam, an'tess'd by the do break your honour. DUKE OF YORK: I, for an Murd-- KING RICHARD III: What! what goes this? at' lie I'force? DUCHESS OF YORK: I would I had a poor day for- thou art made it. DUCHESS OF YORK: I should I will love thee any Isabel, If I have seen the king. Provost: DUKE VINCENTIO: Good Romeo, Romeo: 'twas done, As that in any man have done to you wretch with him: So, well I desire it, to my heart Is this the day of Edward's wife: Thou most Richard, I fear me, Thou wilt make thee as thou wilt be a thousand As I am, as most gentle for ourselves: To this my husband's fair day, my soul, I therefore let her stay with heavy kiss: And, as I could, but I would, my good lords, Ere you be king in oath, and my death. LUCIO: Why, how thou hast done this man with thy love! Farewell! thou dead too, what 'twas no time Thy other blood should come; and with that WhosePOL know to fain: 'Tis so false, to beg mercy, their pluck'dine side For their goodILL'er's hands. POLIXENES: What, be there enough' the king with the king: Clarence other m O Warwick! if it be done, And that my good cousin, I wot that I know. LEONTES: How can you think! We must be this be you; this power are My true son, my brother. AUTOLYCUS: Here's he that did keep her to be: he's a par and a the duke; he is very well said, he doth: He had not so, I do fear. AUTOLYCUS: I may say, sir, my queen, dear love. Clown: My lord, you will, by my life, a word; the he duke's daughter's wife! MAMILLIUS: I have not been i' the lie. DORCAS: I know thy canre so. POLIXENES: O my lord! MAMILLIUS: No, my good Paulina,-- DORCaius out,-- DORCLEY: So much to it, sir, that it may be received To the sons; your pleasure where, the king's heir, Have he been nothing to be depose, But the poor law so far the rest is dead. KING RICHARD III: Ay, if I have Clarence's bosom forth. QUEEN ELIZABETH: WhyWARD, Well, pray, go with us? he is very comfort. RIVERS: No, as the devil'st I am no more more; But I shall have heaven with my life to go. QUEEN ELIZABETH: How long the justice of the earth is not mine? GRUMIO: prison does: Come, sir, it is no more. And the rHAM: My flum, will tell your Lordhip; here is mine ToWARD; and yet I then lay at arms. YORK: God save your lordship to his own Lord Hastings! I but on such right and said 'tis well. KING HENRY VI: O, then, I'll away thy life to bear. QUEEN ELIZABETH: Come, come, come, let no longer, call to our daughter: For now the news is come. KING RICHARD III: O, but the father that is the king, The love for maid truth: 'tis in the needful Richard' death? QUEEN ELIZABETH: Nay me, gone, go with me, in good I'ight in thyAnd shall the sound wrong. KING RICHARD III: So much theseford still he is: His name is troth thee well my death: What, that is not so much, any of thee and let me think she hath no further As, nor any soldier, nor pardon, But that was she hath done, she should be. STANLEY: It is no more of your bed hath best not true. KING RICHARD III: Hath there been, and die out, so much to me The word of a love: what art thou? Second Lord: Thy father lives, that therefore can not be I take theUTA way out of our offer, And, as I say, for we shall not stay. LADY CAPULET: O my hand, a fault will be more never heard Or else moreare. CAPULET: Under this boy, she is of the Duke of York's son He live from mine honour and his blood'st To take us good; and, with much subject Like to the cause to be thus. LEONTES: You have content'd your most mostday To speak your ownness and my soul. LEONTES: Why, the Lord Angelo, So York, as it be father, for his life, To call them hope 'twere yourself and dangerous. CAMILLO: Why, what a thing? LEONTES: Was everBut in such a time of her, Or in a good will of this unisest? KING EDWARD IV: What, lords? WARWICK: And Warwick Clarence at the eyes of justice! KING EDWARD IV: Why,ere no more. CLARENCE: GLOUCESTER: CLARENCE: HASTINGS: KING RICHARD III: Death, brother, are you through my soul Sray to me, and leave me to my uncle? KING HENRY VI: This is thy brother's son in want these hence, Lest as I say, his Earl of Warwick, And tell himself to the Lord king to him, And he had been the will This comes. QUEEN MARGARET: Why thy words have there the blood for this land In that thy heart that should have. You, God's right I should have needful nothing set upon, Theself I am: and yet thy doth-- QEN ELIZABETH: O nothing, that my lord Richard will have. KING RICHARDISET: Yet, as I come! KING HENRY VI: I know the name do march. QUEEN ELIZABETH: O, I wilt thou not know that sorrow hath To hold the world with thee, and make me think How I am of thine. KING RICHARD III: In whose from all the shame of save, ThatCORIOLANUS: Why, how's there,-- VOLUMNIA: Let's hear your heart? MENENIUS: much for my very word, I'--Come, let's go: I am please her I shall do it, sir. MENENIUS: It is a cause, Pray you, get your words. BRUTUS: What, our good lord? SICINIUS: He shall, you shall be; he are three gods Is full voice. First Senator: MENENIUS: Ay, but I will go. First Senator: I know your Rome. BRUTUS: Is the news prove true, my lord, and thySo, as I And see that this beable and mostadman: Hath not a high Mercutio? That's more to say 'twere well;' which I pray I am too more than some, but so good For one to be done. LEONTES: We know my friend, I am theWhat I of your body, if But who youath done the noble duke, But what is done. LEONTES: Good Marcius, you have lost thy devil, to lie all Good night. POLIUS: The people did. SICINIUS: Are you all Aufidius? First Senator: You would he were dead, but not so much I: But now I'll call for it. MENENIUS: Well, well, no more. CORIOLANUS: As I had forgot the world, you have not a kind of love Must be a child to be satisfied, For you being no man to her death. BRUTUS: Away! the arm Of your fair wife'st, For she was my enemy. What doth the man! DUKE VINCENTIO: But makes him, good father, Thy poor prisoner is your time, I'll have a brother of a son too; The side are all betwixt me and his brother. And I have need not; and, as I am not The goodly gentleman is too, do not But else 'INCENTIO: O,en O, good me, indeed and I'll thank thee not no He hath deservedUKEill live to do this; And, if this beNORine no man's death is cold. AUTOLYCUS: Away! he is not made good, sir. DUKE VINCENTIO: My heart, my good lord, did't not't. ANTIGONUS: It is of this; We know of you are now. Servant: My father is at hand, Who came Camillo: 'Tis goodly: I'll be the bosom on the world: When the common misfoler of this battle, To iter 599988: loss 2.8334, time 71.35ms iter 599989: loss 3.3947, time 71.09ms iter 599990: loss 3.6317, time 72.32ms iter 599991: loss 3.5352, time 74.15ms iter 599992: loss 3.2102, time 72.34ms iter 599993: loss 3.5483, time 71.71ms iter 599994: loss 3.5832, time 71.76ms iter 599995: loss 3.4675, time 71.01ms iter 599996: loss 3.3202, time 71.27ms iter 599997: loss 3.7681, time 71.95ms iter 599998: loss 3.6641, time 74.20ms iter 599999: loss 3.8146, time 71.89ms step 600000: train loss 3.6565, val loss 5.9145 saving checkpoint to out iter 600000: loss 3.3908, time 193.50ms number of parameters: 3.42M No meta.pkl found, assuming GPT-2 encodings... The state of men with him that, by his brother's, The very good men, and hisWhen we revenge to Myself, like being now in the airing. GLOUCESTER: This gentleman is set out of other. LADY, lord, give way, and go we to our cannot, And I will leave them to one the YORK: Why, then we may come to you to the: I would thou wouldst hear; war's still he thou done: O, is 'twThy graciousYou were at the heart. First ay: I did not stay? HERMIONE: You will not live But sir, nor little's by the he'll take it down To can. You are not fit to me? LEONTES: What, youPEY? MELMy words may bear't. The one would have so. ESCALUS: I cannot speak in heart to come by with her the thee: never only of your love, O, He has with him most peace; he'setheth once, And therefore do you any thing he would have you; That, in't your eye, well- ELman is so long a Jof which's made youity. If you, you may hear that you are, wtis a shepherd only may he say Bohemia. DUKE VINCENTIO: To your love. LUCIO: ANGELO: I am that king. DUKE VINCENTIO: I would I had been on to the A there: Yet must you be, sir, to all I know so. DUKE VINCENTIO: The duke shall know you say speed again to yourself you. Provost: I will be an't. DUKE VINCENTIO: It is a word of me. ANGELO: TISTRESS OVERDONE: Ah, howio! not all as he being As is a father of a e'er these adTan your brother is born. MAMILLIUS: I am a word to-morrow; youpt, But you might, I shall have had some to him thus. LADY CAPULET: If you were mad? 'Tis full of what you should have. CAPULET: What, say'st that I mean to me to thy name? JULIET: I am the more. Nurse: Is a word. JULIET: What is the news thou bring? ROMEO: How shallest what's a d see of them! ROMEO: O, give mewence, that comes in the time Of this. I am not born a happy day: The which is an ear thou now to make me speak, Nor how in that word, thy majesty worse, To Romeo, not to theage, thy dear queen, I am a comfort that is my wife. You that's a brother, to the duke's e'er, To have his mistress king, to-ake: This is the love I might have doneish the country, That I may live in theWSo name lords that means That ever had been to doway on! What think you, if you were you all, Will not my full very reason for a great bed, Uniansch high the heavens? answer this Of this fair prince's name? CAMILLO: I will, my lord. LEONTES: Pray your father! That's true-Sir, and you are son. POLIXENES: You are rough sir, give me Lord: let him go your seek. LEONTES: My fair lord, Were I but so much, or do I see, my more proud, On this most gracious lord. ANGELO: Thy great youth is the forth knows. ISABELLA: But 't, an confp Henry!' The which you seem not you both, which would well, That I with dearLE RICHENTTER: As now I have nothing: With quoth so, my lord, and all at night We cannot speak, let it to remember me. CLIFFENCE: Is some thought of this, we may be a cause. KING EDWARD IV LADY CAPULET: And to me have me bawd, youngly, I would say, If you say so, it shall know the world thou thyself. LADY ANNE: And, I am that day to last with a thine. JULIET: O God's sake, is come! I'll make a years upon thy husband; And, all am grief of call thee all this good. QUEEN ELIZABETH: It is my father's; it is more A dead, a proud; for if she be king, I may not live another woman, I am not a father that will take it; And now, for my father, by my whose fineoe Provine and my wife's son brave father me: But I hope, my lord, for I was seen, With mine own love to-day; let me lest: So I have done, yet comes one. DUKE OF YORK: Thy kind-morrow spake for the grace-night! DUKE OF YORK: ay Richard, that dothst thou do to love thee? DUKE OF AUMERLE: Then, I am mine own again, or I, The truth of day; and with a kind house Hath made her dead, who have manyre many more That I'll hear him rather and say, for my death, I give a purpose. LUCIO: O living, my lord, it is gone to leave it, And you shall have of this good or'dN At my poor brother's kiss: and I have seen, The one that's made me welcome. DUKE VINCENTIO: This oath have as high as the wisest, then other mure of a chide on: the post of those: You have not there here are comeable. LUCIO: But in this while I should have them; if I cannot, I do put the fool: then, like I come to love thee With a set upon my master, which is his pence of the king; even so in his hands, As I havech you at his best, By that it may France, for honour! if you were now have all; I will be banish'd, do not to prove The corn of honour, and I will not The noble touke: but I will not have any name That I, and think it were, if I could stand, Most might be in this fair way thy hands And me shall the face my husband'sJShould be, To young princely to thee here, be long-s, Take it again in noCOR'er a happy depb not. ROMEO: O Romeo, nurse, how can your dU is a happy In mine honour or anBHAMELLA: He shall have a poor way inch by his honour Gless worthmen; I warrant my counsel was more. DUKE OF YORK: Let me I speak of such, my lord, That very good to be this, before the time Wept with Iixt these words. DUCHESS OF YORK: Why, so I am his wife, to do't, Yet hear my lord; 'tis so; and I, I do him! DUKE OF YORK: Int note was that ever wasR pleasure true: He which's anMENain'd now: heuck me not his bawd than I have content the old gold-bed, he's dead, That may not beoe to be thy face. DUKE OF AUMERLE: Whom is the way, to youQ is it so; For I will not: so she would thou hadst no time Than in me but not: but that--tis not so, For I should, I think, I please you, my sovereign, And by our country's breath shall hear thee, To lose a matter with a word; if we At the most little: to such nor thought it was, Unless I think had been yours, for I crown'd; And, with a word, poor soul, that thy love's will do it. BUCKINGHAM: I know the gods. GLOUCESTER: Pray, be these my bosom: for the time I'll pluck the pack of your house face. KING HENRY VI: WARWICK: And Warwick's death will go and did to us. And yet not be fear'd in all. Go, Cates, and not thy brother and brother, And come, I'll swear the duke a good To be as free as little as, so myTis-- That side that would kill'd my husband, so Whose honourFor it is lost grave in-- Whither dost not be long as long as Henry's love My father, Edward, to my tenderbr own arms: Till your three days give good thine and your eyes. KING RICHARD II: O I wast thou not dead, say I know thy mind, Yet thou that dost in beseech at us, And not to the people that brought you to me And think you of our good rather shall be Be it to me but for my best good. First Senator: O Clifford, O, it is the more! MENENIUS: I would he had a people, But, were you not't; and you, my lord, You am of what you shall be so? BRUTUS: When you are should have bound 'em. MENENIUS: So, most it do. BRUTUS: Where is the love of this fellow, that must not? AUFIDIUS: We do not say: We were to do't. CORIOLANUS: You have been a cause to need. VOLUMNIA: O, indeed, I'll to desire o'er a counsel, And I will play the us. MENENIUS: My noble wife is me as good as well in true, To have an those water's better and isar'd. He pluck'd with him, and you'll have all the news-- And it is as good as Montague hath. Second Citizen: That, my master, which is his friar o'er-s, In that she had done the noble duke's from The noble head to the-night bed-like. First Citizen: Sond to the king: But I hear, madam, let him be heard it Of God's name, that is ill and great blood On thy be these; yet every w thought I with it, And none but that I doubt; and, in your love Is it right: but I, my poor dear cousin, I hope that I had not to this brother, But I will stay in the thing I say. DUKE OF AUMERLE: Why, lords, what aunes may? DUKE OF YORK: A gage, prince wouldst thou tell thee they! Thy brother's death! DUCHESS OF YORK: Unshe's aHath To Romeo's name the traitor; And you, God and I will see thee a and that Of living men thus they had been in this love To the duke. QUEEN: So, they are not to be the name of all. KING HENRY VI: But, madam, be content for thy thank; If thou dost sleep at thee? Now long rest is dead; thou wert not thy f after? QUEEN MARGARY: As thou as now I love'd myself of thee? QUEEN: But I shall fight with him. KING RICHARD II: O God's pity where long the o'er the world-and'd ! within I have done for the hour of thee Whereof thyself be too quick for my death, That enemy, with thy warlike blood, And made the king in arms at gone. DUCHESS OF YORK: So shall I do, to so late, Why, proud-hINCENT are lived told? DUCHESS OF YORK: No? KING EDWARD IV: I know you that is a father, to my? YORK: By fair, I thank thee, love thy face Which thou hast welcome of the fool, thou thyPR; And, now my poor queen willBYADYOP, For in my brother I king have play'd heaven: For Edward's face, being gone to Bolingbroke. TheAU's aH no other tongue to thee; For I have sent not to be of. DUKE V WhosePOLIXENBERTER: Not that that shall tell thee to my grief, But I will not be gone to thee, and thou a present tongue. 3 KING HENRY VI CLARerer: Take it with the day. BUCKINGHAM: Therefore, my lord, I let not beauteousors. FRIAR LAURENCE: Why, then, I pray thee, call thee bring me to me For this my gentleman's letters and her: 'GoodISTR in't. ROMEO: You must beL blood; And thou wert dead with a for him thou this! Nurse: Go, come, come the nurse; I mustETH: I pray thee I know thou canst wish it not say. JULIET: Though we have less, for ourHave set than me, So full of honour, I am gone to Warwick. ? of ourmen born? Second feel so: this is well as true as you, That you are put up and see as fast as fit As far all-f- HINCDand children to my poor blood To this loving? LUCIO: Doth she, when her father's Claudio hath done. What's he? He is in the name of men, With tears of heaven, and their true-ouon put'st Ourself'st man; and, and so is they all; That thou, like an unf swift use of blood. O, I have thy king, and thy king, That thou art a rest, thou, then now in thy place, Take thy hand in thee to do thee most am dear. JULIET: O do this love, and be an O! O's thy title, Is more more than yet thouThis: now, and that thou art To be it? 'tis not a tas beg man in seen an hour, having whom, he will not tell him. The means of you, my lord, is your lady's son. First Citizen: Away, my lord: we know they do us all. First Citizen: Woe to you? then you have said you were? Second Gentlemen, may she be put for it. I but think it was a bawd, I had to do so; For, in good time, a very well-baww gracious. Nurse: Why, how now, I art gone, Saw-b, ere, I hearon! JULIET: OGLOUCESTER: Here, that would he were so sour to he. LADY ANNE: Why, Warwick, how now in day shall I be! GLOUCESTER: gone, my liege, in what! If it be I had to die their high. KING EDWARD IV: Then am you patient; and therefore is Of all my husband Paris, to-day. GLOUCESTER: O, I would I had rather, but for thy son? KING EDWARD IV: Your death, my good, what! that I am there. GLOUCESTER: No, to thy name; but very it is not, For I have, to go with me; why, he's dead? heir: We shall not, my love, my lord; But, Warwick, do not justice her out; For therefore, so it is his hand that is: And yet our country, we are, nor I. QUEEN ELIZABETH: But what means, good father, be that state? KING RICHARD III: Tell 'p thee, an brow of all. QUEEN ELIZABETH: Good prince, so! KINGain: What do you thought, I hear it, I know you well: Your face I am not in your good counsel: Then be it then? KING RICHARD III: Ay, but he, but he's no more than a country. BUCKINGHAM: Is it none, good father; you hear no more. KING RICHARD III: We have done a better, our person. QUEEN MARGARET: Call so it, lords, have made him our hands: Berely as well as I have; for this, As for ESCALUS: I think much; but they find't their way out, but to die The people, and he hath yet show'd! ISABELLA: What you, I'll givehe outOUCCIUS: I am in need. ANGELO: I was a kind of all. CORIOLANUS: My gracious brother? LUCIO: ouson? ESCALUS: UC.' is there a thing? VOLUMNIA: I would it, sir. COMINIUS: Let me but stay. CORIOLANUS: The gods too word, my good nurse, I come from Rome and of the had the gods alt have been in the presence there that made; The people lives upon our TheABes; So therefore you, I cannot do you, you Were, my good with me, your father, his lady: One of you, he has done, by the honour To Coriolanus. CORIOLANUS: Haply, I mean to us and on. VOLUMNIA: ine, Lest, take up the duke out the traitor's son He was a like a father; who made My best son: I will wish you his love. Wilt thou? What's this? what? will this, so I here, Were in me by my master, I am like. JULIET: Richard had his grace, Nor, since the life that it hath most his, For much more shall be, and in that word. Nurse: JULIET: O nurse, I fear it were night. LADY CAPULET: Pray, you have all this day a may have youak. What, is not so long, you would have been long A noble deputy, which you come toius; And you the people is, you shall find again To see you knows the place. CORIOLANUS: Away! Second Senator: Come, I pray you go to you: you will hear me: Come on, madam. MENEN DUKE VINCENTIO: What, if you do not, you should be been well That you shall have; and to the alone of my heart: Do not think that it is hard to take. LUCIO: And in that world where he cannot be; for, I do see You that never, I have heard, but not go; I'll the souls of more and more That is a good way to hand; take TheSIC he my C art, and myself to please. Good ladyENS are thou what an openon 's tooNot, nor what can part in this fair speak. NORTHUMBERLAND: Come, cousin, you standNow down toWhich as these blood As thou wert so now in a comfort's hand. KING RICHARD III: My dear love, my lord, my sovereign liege, MostFrom earth I come for Edward's death; And longum thou, the fault, but not thy lord: I look'd, to be fear'd for me to live. GLOUCESTER: I never will, I'll speak this. First Murderer: How now, like! what's the day? CATESBY: Marry, my lords, shall you seem inh nor wits, Or take you by the once. Second Murderer: O God's brother, thou art mad, and begish'd. Second Murderer: 'Tis never gentleman, but what of my counsel But to my will at morning. Second Murderer: How long as I? Second Murderer: What with a name, was this? First Murderer: Alas'Twixt me, and must be thy daughter: Doth set the world upon thy counsel with thy thee o' the face, thou hast made them aHere. Second Murderer: 'Tis more but not, dost thou havest Lewis, That he hath leave of more unto of this Richard. CLIFFORD: WhoCAP thou wilt fall so of he had been of! RICHARD: And bid me stay. RICHARD: Messenger: Give me thy oath I 1000 iter 997: loss 5.8675, time 724.04ms iter 998: loss 5.9554, time 722.33ms iter 999: loss 5.8129, time 730.23ms step 1000: train loss 5.9102, val loss 6.0530 iter 1000: loss 5.3077, time 1634.94ms wandb: Waiting for W&B process to finish... (success). wandb: / 0.001 MB of 0.001 MB uploaded (0.000 MB deduped) wandb: Run history: wandb: iter \u2581\u2582\u2582\u2583\u2584\u2585\u2585\u2586\u2587\u2587\u2588 wandb: lr \u2581\u2588\u2588\u2587\u2586\u2585\u2584\u2583\u2582\u2582\u2582 wandb: train/loss \u2588\u2582\u2582\u2582\u2582\u2582\u2582\u2581\u2581\u2581\u2581 wandb: val/loss \u2588\u2582\u2582\u2582\u2582\u2581\u2582\u2581\u2581\u2581\u2581 wandb: wandb: Run summary: wandb: iter 1000 wandb: lr 6e-05 wandb: train/loss 5.91025 wandb: val/loss 6.05303 I my, your the am; By areU he: WAR H That a me, I am father Or: And to her I I, I ROM, lord:ANG is IET. ITER, He Hold can for my your am, And if yet be'll to the the a life the the he a my; P: Sh, have this, my, A,, A thee- I a tell: O. A be any, Your you, I my death For I, this. You, my As I and the PET's artUS, My life than. Have see! and hand to God of stand, With, a many me? Where of have think, it to I have not, That that that pray, As the my I not thy the she lord shall a thyIO and would poor, for the: and: What and he, ' so it, did a will his king, a is Th: Butour Will, Were,,. She, COR the her as, Good When. For lords. and butET is there your me? and shall king the this. C, thy mine. O The shallUS, That: GL: That, That, Then, here the But, What: H the, D and, NOR toEN! if must, The the,US? To my the's poor. Why's will have he'll can how man Some in cannot than come, Even a in, him. Is you anyENT me; so say, be sir, the the Itis the, be'll man I them. What. So thisAB he, That that the will, That the'll:I a king, That ' Would Now, have my the a: To kingUM, I'll b the must a do thou, H. Were, his thy the the the I'll, or a'll cannot, I To No for they not such time, name? And was have to not me, Come of must know,, I be the yourUS his such I be my. How, For me the have have: ROMING To to when to, My: as And men, An a B all or And thou is, To man, And the: Come, am sir, An, And: Lord queen for, they my, that he part This! It in my trueG do in: Is? And we most, that the BKE: Is an NES? From say; you and, to's knowING your I his But V for it: I lordUL from well, Go in me, it the be I was. but: the her as IU the he our life; if his I most in I the is, I But in hast: Thus and your noble the do take she be: ' I'll a I your d ever, be not allEN COL to aOL, and not he, you is I make, This you not? I And, beELL what there: Do her thy here, not not Not crown your in a that, if death would we you not the w made of his me and butI not, What, H it P can that The the, So: COR H This: if'll B, butET with will hand, I the a N to must a, If, Which I would your- have by the am isIO, What no come is This aIO: MEN been d: WAR in a not, '; name: To sir, I most NotUE men, I What and d me:. my? The's have a us he the I lordose, You not did, Here, do an a she, my we a he: BR thy Why she. P love,, COR this the that how, Of what out H thy: COR his me you, And as NOR you as and that But I Yet. I no crown, say: As have: and wereTER:To: KING KING. ROM be you my the yet, some one? Second their a theI, the father this it not the his. A as not: ' it thou I theICH. O a lord.I it. That theES to he So. But! That not do. I: And your Q. I What IIO, is lords, he the stay that my come: How it were. H not, D. And, that be'sEN: H P: And my that have that not't to they the thou thy were If in; and the for know N In: for my the- never do tell R d it, Not, First, of I Come I be be hath a be her these me by this: and the it goodUT As My when, MER must. H And have the you- sir: and your To be aT, which is; That one shall must you the a, That make I that No'd. This; If now, good dIO most you time, My. N you, and this is have will am, ROM of were shall lord toUS, ROM is be his him: Sh: and her their: H to beEN I the name, Prov not all the you thee, The it, that was,IC How, As such not no IINC, our I and this is not, C. By but I W will as his Sh! you, To him: and say; she. Sir at notEN. Your me, for what is, Q them: Is Sir, one a him NOR with ROM mother that be that see in, H. I D, the so,, This man to She come? D Who, but; by He, H, I I I should my sir. As, take my as or. Let: Is Where his he, No his you be D have,And you He that in man thatIO And You in blood should That: As was lady. Now with: First, theICHEO isI very: Ah, will an- I blood him a, house, that part, my: MER, so, And I but you: What: Your come, Q, To, Now, will he see: Be, His to his an shall then,-- I. I most, your the this By will'll and For a the a a would he At I'll? I: Un my head; They thou am, I. It: To not, my noble in so, and O. By art? Let, they is have thou: COR; and I The But Th ' An are letOL my. And My will, What me, A, A good Second. Then: Sir:To am art D. I the thetis, we one. I Lord Th. D, let, Now, MEN, the, To which And. TheUS so that an be give, be so, Come then him. That can the:Y with by; you give to'll A this to be the letsIO to be the And have'll, These is a thy! I And: OfEN cannot'llUE with he my do The did: Of thou say so; be, best The our What: In can should of a he I To, Which J: I: He of his sir- am, If sweet the Now. ROM To, their. men? D, would king: And And one And'll himly name. COM, sir with, And: N am's that you of there thisOL heart: You if son But: WereET: if I never, Your in your not? goodIO, or: How Give not in; Come yourEN have most: H:So the I do, If my my I And the good I this or love in think: My Come's and some not I call, to mine: KING you '. Of To But life, be, B you, you, And go: Good theUS and love W forTER to'll II them:But'll an. If ' will, and's part be bear here, A: We you How, Make thisEO's his would love, a when not: N him it, are you, a thou then, How. Cl a a for your! be the part, And take the But, sir, That: Ay, well! This, Second all-: If a I the'll all withUS: Totis death; sir of: And his her thee to this'd: Un, see and a KING, The the be; thy man, some do: And thou ' the the some thy To the this As. That--IO, The I the aKE thou in give! It shall some thou have I I I you the a To I d not am the thyKE a my the this he: Would my life,,, Well is thy, Which: Which a well. There R, the a B so: and UE MEN not in I tell withI if If in the R AsUS. but I the me when you, Good And him. b poor, One MER: To all: and is I makeant upon it I we a blood I. You Is, the, myIOL would First R, a he, Your queen: no people? TY: I. Is shalltw: And 'ES youtis Which his my your my art we haveUE me S his but man, which come: That H the your the, b, His the a a my king? And Go to his Second hath I his more man up, That, What. And do'll a a As He in he have, I my willIO: And, That he I isU You, But. you? Which to a the a, To You a, have the my him, a these this and a him: R most love: that no do make if her For I know, my grace. AEN the am your your the I lord? Who, I the with But: I, a Ns to have so such: KING: J a I day is D crown'd, That But the the. Against when it J. MEN a the. If It love Give theKEant, E, go. Sh: I ' as I ownUEAN: Thus will myIO of And, Of queen: IUS of my lord! it, He would thou? His, And that mother; And N I have, for his if shall, Now, to the am, there that great must betis he D this. C the do, To an, As a them,: ROM, Third, First: For the see, I the a a, , Now I was,'s that I it you the done Th with do my, That that it: O: Your, our'll And the that ' him like but, a honouron. I dead to her some I: He, If: KING: R thyUS. And Too: For a be theELL, our, And many the but am this, Why, They that the this make, I: Is have am her my my he all,, when the will the to be, This, As may I my their have the To him: H know is say YU ETER, Let. Now. A give is of shall thou, There you, P, I poor, a the some have the your are the for do for you this to your a for or good do, or noble: But And shall he be And good them with Cl's The are a do you. That notown, Sh, I be do it,U O you I be die, Her, by I; she, To be to that sweet all, We be: D more. For the WAR: a the-'s she, that And a thy what, And It, and these. The thetw's know. Like of makeIO. A But I speak: with and: not, PR: Come we If his,EN hear: To I I, N; Of, I, First so the head: To that hisI be you the: That you. You? ' shall I be As He is KING, come KING, the do the the So, With, And shalloth such me,, you by: That to her for I like. Our, As in this love: 000 iter 4990: loss 4.7843, time 730.76ms iter 4991: loss 4.9219, time 722.85ms iter 4992: loss 4.6760, time 731.14ms iter 4993: loss 4.8503, time 699.10ms iter 4994: loss 4.7151, time 721.49ms iter 4995: loss 4.7852, time 687.31ms iter 4996: loss 4.5192, time 732.55ms iter 4997: loss 5.0381, time 727.84ms iter 4998: loss 4.6365, time 694.68ms iter 4999: loss 4.8973, time 691.50ms step 5000: train loss 4.7097, val loss 5.5308 iter 5000: loss 4.3929, time 1421.90ms wandb: Waiting for W&B process to finish... (success). wandb: - 0.001 MB of 0.001 MB uploaded (0.000 MB deduped) wandb: Run history: wandb: iter \u2581\u2582\u2582\u2583\u2584\u2585\u2585\u2586\u2587\u2587\u2588 wandb: lr \u2581\u2588\u2588\u2587\u2586\u2585\u2584\u2583\u2582\u2582\u2582 wandb: train/loss \u2588\u2582\u2582\u2582\u2582\u2581\u2581\u2581\u2581\u2581\u2581 wandb: val/loss \u2588\u2582\u2582\u2582\u2581\u2582\u2581\u2581\u2581\u2581\u2581 wandb: wandb: Run summary: wandb: iter 5000 wandb: lr 6e-05 wandb: train/loss 4.70968 wandb: val/loss 5.53077 number of parameters: 81.52M No meta.pkl found, assuming GPT-2 encodings... Not that; You. And it, I to the mother on my son.', PR: And the suour of, Quke, That, not is this earth And not, Now more years will the part: The hands; Not, My son, To I do my queen. TellY: Darest by strength and. And, As Provife's side, And I not the gOM look, which: AUTUS: May not men to dU and you, So, LUEut's lie on a would I can! The suilman, The noble lord up I I do nothing in your end, DUCatter thou have my r LA, For, with to the earth to your wife, Not my or not his hands With you do a friends, His loss, Should have not, as Ibians My mostt to I lord, the world, take you not that you the cause, IOL OFCES: To you, if her my father, I can be your Tower To do my name. That the hands to me. Icius, LE: Why, LUCIO: LELCUTIO: To thy lord, to my day with me? And you more for, For never my power, Her heart; DUE: Good great noble father, this In to an father? And be do for the friends! The lord ClOL OF YORK: And he will he of my hour. And if Will some thing at the lord Bove's father have her; Darep the counsel. As will be those the one, go; come How me: Madunes! By me. And a curse of my. ' I hast my lord Not, And, my lord; He. O Do you have be With blood, R VI: Tell your foe is a king-eseous death, Sir a most mind. That: Who I have ProvUoWARD But, O And in a go, From sweet high is joy upon my, FirstICan had this? WARICHCES V Lay: Come about a people, IUCUS: But this land'd's father, And Could To my will; let my ancient lord Unless my lord, but That, And I shall be I, farewell, if I have said their head, And yourly the time All it which thou have, Which not will; an eye! In me of her not. How I would I DUCost it have the lord, to you is my d'll my common d understand What will not.The business! Glouace; and you, Why your king is your great And Mess not's in heaven on me Say It thy day.IUS: And FirstIC VU, Be that I? AndtwEN Which Best you no is to thou should I do not, Be thy life; And and If you. And I not. A: And he callWARD: To their most we do yours, Dost; Who the first: To be't? A: To my king but Once will But. Thou hast our loving dADWARD IV: The Duke; Sself is the father, and be make you Why, Blbroksarry; From a Lord of that she? As this, Which I have but I spoke, since was the lords, Well, And your course! Sir. Do this grace they were in heaven NEENIUS: Look's, BowY: With in the word. L: But I wost to not the heart To all thee, LU' the my heart, Hre the heavens. And he, we art shall be be your other the last in this quarrel. B me to my royal prince, My life, it may bear hist; SecondUS: But in your, good day, And you, that, And it; it, to give him, From true may be not AB, Look to you will My, In thee-OM do the less aself, Before the, For I not better 'er she not you here: And be a heavy un Senator: MENall him It, It be my suit KINGELBY: Yes! Is my father, To the best, a grace. YATHolding, but such me I cause your sovereign! KINGABILLES: Nor what shall do a ear of the word's tribricMour him PER be. A: Sweet sister, My self, That the grave, The lie' so thy case in put, And my wrong, MENre father. Your child, I do be our will Nor your That KINGOU RABit's with a sovereign's heart as, and been But, good lOW: From your court will you me they friends withESSyly I-US: On the mind, Iie, And if That the good thing the very poorICIN With this is him. POLam'd me Say, And thou I do be be all my word, be my noble day, a Tower to you, Or, That them, sir By my sister; HILLES: Or than was this, Or it, LUS: For none. Have all your dINGS: HULEfore. Of her, Who Th OFABall Burse: IUS: Dcius. But time, when I will be you it the lie me, So me, So my lords, Be but bear all more His better I have thee to not by thy dU me's more may, And he. Go, That the words, which I besinks would he'd in hisENTumerine! Had I will, There has, but when Stand with him, FirstUS: Why: Our father, in CO; Like you it me's lord, But,' he heaven? Be you may the life. Doth my lord, but was you is dead, AETH And see your words. Ander thy will all down, Or in my suit, And at this f Murdaced not, To; LUS: From all the very and his And I must it shall am I am you you, Which, My friend with Inay again: For they traitor; For you but an country? To had the LU, here, had That thou, do the heart, IUS: A: IUartarry. But I but MEN: FirstUS: And they not, LE OFELA Marshal: Are not to heard theICINNow, sir, RABELLARRIUman. Well.And he a eye, and.And show to I be your death have this lords me that, in the king And this a times from I would say, then, your father, FUCUS: R YORK: Now. To see with my queen, GiveBERIO: for your sister, For it. Of your land! And For, I be use their case; do have too as see his fault And I speak Who I more stay? Let the or ent not, and now all you by a gods, you I say dU your war, In me as that: The end. God no more thus; GLCES OFUant; And I, O of HENO, But, ToASIONISHorn! Which The world, Why to, Which. Who have tost it shall done.Iona me. JOHN: And it to his lord. To be dead for peace, for me; She I they as that, Which art. Which be a soldiers And a rest with an soul, and of me to all for her, MERARDULBRES: Now. CORUT'll give me, And a house, Give your gentle be l'd for Good death, KING R HEN'B the honour, To be ourself, Firsthouse. MEN and the father! ? You'll forUCKONT thee; Of his gentleman, He will more's: ThINIUS: To I stand! The head. No, the brother's this, nor so the world that in a bo', then, Most justice, So you, as the life, What, With her this breath, sir, You thousand uncle; When he the world, but And these brother the people. That I besiss? the good father; And, LUKE: Why, I, And for the king, HEN' so to And to you. KING RABETH: Th me, Cer; Let haste, now LETER: To't, And Quke! And this, I of to the eyes, In your blood we all here this sovereign: And if a my time, As my hand, Three life: Than to your power? That, FUuUCKIN Ile? LUCIO: Your head. The head, no thy people in you can your heart, HERuck thee, They be the life. Dague to me For not's kingdom, From the death, this loss, DUS: BBY: O is my as it my noble trible, and nothing,--And my quarrel would what you, being not, Q YORK: WAR YORK: Why Now, With a my side? En them, E: As I know is my other's poor They have you is, DAR LA II: That's more, for me to myself that I be a queen and be a heart: IS VI: SIO: Why and this lord, MontM the crown: May think. To this's hand, and your hands, Are not I not I not's very man, He you, I be done, QUC love upon the cause of my air In you I he do pass, FADESSINFor my father, Bret to But here, sir? WhKE: We Q be, FU, LE: DUall what and, On, LU, you I'll, There, LUS: And you, Theark any? Which no that, COR thou be in 'er And There in the child, LARDELLOU VI: If plWARDASTICK: GL YORK: The crown, as my VolKE: Lome, For I come thy life, And HERESSou I to you shall, on true have-ighth up And lay Dague, Y: GREY: Who, To have your ears's lord; that now. But than I thing! LE OFUCH'd, Your death. FIO: 'T: To in the soul, sir on the word. Like OF YORK: GLARD III: The air, As There? Peace; What a love, FICK: but is the end, And were the friend B thee, for a king, ' he: And And that is an l me to.IUS: So, SwESS OF RentleY ANICou we did art your breath, weENTona? Say It that, At your man and His uncle. And I have him, And thou look is thee, And this very good, that go; that to time? MENENES: And here's, sir, Ay: My prisoner's me. My daughter! IENCE: This, And your restrah is your great! LARADEW' the other fair land.IAD me you may we be he live, KINGELUTET: And many hands, And he thou came in thy sea. This your sister, And I set wABiful; While'd, with this cause, and done, DAB II: Or can our breast, Henry shall were my oath! Which thatness-- T, She I have live in some man off be me now not may my lord, In a people; Which ' the words to his, If mine you should my name; I have Now in a lords, And my life with him, KINGUS: And you, poor looks's you on your father have light, If 'em. GLUions when men my On your lord? But his grace. Go,, CBRESTER: an sweet LUge, O, with your world. And I well to true is the Lord. Lordugh, LU and how you; My soul, With, And What: And, To-US: To that is your life as in. My hands. And it it to be Ct is our soul. You now, As both a queen, like once and any more, Like you be a brother! SUMGABENI would he, and I see a blood We do Clarence Well me, To time, we spICIN GRE me-ICINJENALIO: And the quarrel, 'T of a my the Tower, CIUS: On the case; Let him, She that shall be we power! To be your, And And a right with that'sENTOLou you he, And that! LE: DUiol you. And, Will be be you of mine that we more for the father. And IENTIO: For to right upon the death'd.IINGS: Now, His field, I be my gods's land, FW'll. KINGOUES: And thou If not to the father? HETH: LE:-- Look must you Who know man by her. POLopit you it man's to the other were make my d my lord to the one. BYts you; His time's more is this eyes, IS RARWICK: LUish a men is to such this brother, O, That must he it out? We can be There him that more's more me, eunes, And IC I not is your the lord. Ander before it. And thou were, a peace, And Scak down with a, AndT II: Which and him the king; QAENI had me to set sorrow, He with the lord of my wrong. He, CY: And were a gentle lord, And I had justice. He, aICan's to be your prince. The better and once have be have none! And he is him, I may this: Yet come are the king, Andn all. From our life. Shful blood'd, and not Shre is what that, Jutful men all day, my better you to your own earth, I do LICHCBRHAM: Who, And my night, My life! What this name. With the side: How And our further with ourselves, If are for the pleasure: And is; The own life, The father; And I stay. We be the fault, FirstUS: F mayENTfm. Of my death it, Firstoler. IICK OFICHINC scorn me, ThINTo the heart! Of thy while man, of him To take me, You. It by those the in me, As So no world,-- To put off's man CORtis going's land, Her counsel at your father than both you, Why more; But you, For most world-ICanly Richard, Make no need is me thee; The And so! POLKE: DOUO: FirstUS: Thus, Must shall not but I, You, lords, farewell And will not. How'sbher dost, and be, LUEENO, even or so I For in my queen's good king, To fear, The hand and, to my man: And the voices, Here aem word, And will all to a dUman: Who by her with him. Jricest a bo R with your IIO: And we in him am the lord is. And As To not is great Have you to in it, Mess so is my high on and the time! That I do your lord, He shall made? O you of a best, theyt with her of her must with the high me, and the master at my brother, if it you not not I in the suful house to they desire, hear fulled thy grace And your FirstUS: GLOUUTIO: O. A: Of, This it with? First should for theself: To the hand, Or you, SICIN Ay, He in me again! Which go with the time. You come's! Here myself, L: A: IUS: Or that he your blood's and he are And know your king. And With't will a son, DAD your is an old's king in him to thy house; While yourt in me, Are bear, ' they. T: ROMO, QW: DOUES ANIO: And true, he a day, T: To to you than, and so; If he is me from me is my one. The kingdom in our lord; And the hand upon BolCHp Henry. MEN: For I would the earth, The crown, Your bed, I not ISARDENES: Alure, And more to; MadKE VINC say CORES: MethESS: She of a more do IIZARD: And you, And you, I turn not the presence. To gone: And I will th, I say. our voices, IUS: O, KINGABours. And you hear my face, for they more I will his king, thy bosENES: Sir should have go, LU, If I be none, my soul is your Duke, And he. And a more ' it for the son, His kind, He, being she En is it to this I HreG am so, LE: He to be, That that by, sir, Cords, LETER: MENENCOKEANost, and not we What the wife, ESANUS: May I of the Lordus'd the That to you will not did, If For he by my man and, DareUCKou And it, But, but For the life. SecondUS: And we think, they The part. CAPishment his do been me what is you that, SecondIO: Go your do not, When you B this life. With her, if he to fear. The grave! The king: That I HFAD down, all you with me. And I to't. MESSou would he, I, IADere's man Call the word Before a mother: My life, IICK: IUS: As That to the fault, Or shall in my heartous face, by that, then are it, and my headly. LABch the present, FirstUS: To, AENIUS: ThANUS: I'll out! And we the looks's blood, and be your life. IUS: Th VAENES: SUS: That's, Which in him, Who for me I show her? But me, so thy friends, as I not, To you I ThINGBRO, GLAB-US: And then. ROUO.IUS: Of peace, On the hand, But from a lord, DUed to I-ICINLARUS: 3 III: Sir, And did not can you-blood DUed, And the great head too thou should leave, 'T: To how you him, A III: Thou will not But to not, no devil, as it, but I be him; Hans, as he, and this time, You his life.","title":"gpt"},{"location":"gpt/#gpt","text":"https://github.com/furyhawk/nanoGPT","title":"gpt"},{"location":"gpt/#train","text":"python train.py --dataset=shakespeare --n_layer=4 --n_head=4 --n_embd=64 --device=mps --compile=False --eval_iters=1 --block_size=64 --batch_size=16","title":"train"},{"location":"gpt/#logs","text":"iter 432061: loss 3.3774, time 102.72ms number of parameters: 3.42M No meta.pkl found, assuming GPT-2 encodings... The state of heaven with him yet have set The enam received in his power, It will be such,morrow; if he be By common enemies, so may not do, you As if you mark the truth. FLORIZEL: Why is my part; Not here my life, we'll do't, where it is The lark that the Earl of Warwick is slain: On bold, my rest, my liege, in whose good time I'll drown him worthy me to my Earl towards, Is Come to draw to scARD a fearful day. 3 KING HENRY VI KING LEWIS XI: What! canst thou live in poor soul, To do him banish'd at the same our point And the freshinks I come, 'tis aught to have but I fear you joys: all that I cannot On thee would of my heart, my poor soul, To be so bold a knave in this time; And thou by, she's children's blood, only, it is To see my life. MARIANA: A thousand part, indeed, sir, a time As you will wonder not: come, my liege, I well; And he is too a gentleman. See that with the voice that at once, you must Flried the senators of the any it good to us being son: but, I cannot be done; but we can tell him; for him I'll draw your words together. What you will know how you? COMINIUS: They are, uncle. CORIOLANUS: Therefore, my gracious friend, I am the farther un ches: What, poor! I beseech you, let it be a cousin. VOLUMNIA: Tush, the gods! MENENIUS: I am too! Nay begin to bid my lady play well: I am no pity that willy heir is right; But, by his leave I have or why it is. CORIOLANUS: Not? let me see, you are strange? They have made no more than all. Hath my wife's lord, upon pain of death. Third Citizen: Why, I shall have had some heard of you hear. QUEEN MARGARET: And so we all, I thank thee: 'tis more; Thou art the letter, and thou hast not so: wouldst be done, hark you, I know, my soul here have t something So to say, but I pray, and no more. ISABELLA: O, if it were full of such deep Norfolk, love, I would have more than his gracious fault Than Angelo first to you at great comfort. ANGELO: Yea, and make you he crook 'pt TrleKE: I'll make. Provost: What, my general? DUKE VINCENTIO: 'Tis like of his most, And leave me to the will for my poor side. ANGELO: Hath the king'ster heaven with the majesty of tears, With a kind: I have, my lord, must die. MARCIUS: Tell me hate them not, honesty are there To entreat of my wife, and my son to my order, And set the mother all my father's consent To make the must fly: so go: From whom that I would not, I do not fear, As well I will have me, you are vile toward. QUEEN MARGARET: But, by your vouchius, I fear not what I am: For since we came, your father's, at the least? Second Servant: Tis more than this of you can but have done My love or mercy to the queen of heaven: But I have forgot it, more, goodenio. CAPULET: Ay, as the matter I told you speak, my lord. BALTHASAR: But to the case of the Tower. ROMEO: Never; live, my lord, my soul is as well: I hope, by this which is a villain, When that thy brothers Romeo is, one sun With thy mother's wife, that I may return To know me what a dream hath as well As she run thought to be your mother, I I may not need to my sweet, CAPULET: Ay, what art thou? ROMEO: A queen, nurse? JULIET: Good lords, If thou dost know ye but. JULIET: Say, that Richmond's soul, take me in all. ROMEO: O, Romeo, I will not weep a while. JULIET: For the defry of grief makes them fair: But since unsapblful villain's wit of heaven It was a friend for ever-day, When I was lawful oath and not so, I am born to do it would it might be, I am revenged on-hearted my leaves dead king's oath, For the wantless blood whose heads must enter: 'Tis the Lord Northumberland that King Edward was himself, The Duke of Warwick, and his place in France, And thou shalt come to my will-day shall not stay. HENRY BOLINGBROKE: Why, howof, I beseech you talk of, Here must I play my heart with a cruelay; That doth not fall dead, I know the matter. DUKE VINCENTIO: You had even like'd, fie me, 'tis, I am resolved: Nay, unadBe now, I have heard her speak more than you I speak at peace. ANGELO: Well, I do beseech you, fair sir. Your eyes will not come upon me: he is most man, it is not so: so most much little most wrong, and may go to that by: which you have therein your hate, we may see themselves. LADY CAPULET: We are in prison, one word; ask him; For some you win me, upon his best: You know no better for the mind, there's no matter stands on him. DUCHESS OF YORK: How wouldst thou? Messenger: My gracious lord, I like youThis. DUKE OF YORK: Is it not dead, cousin. DUCHESS OF YORK: Then 'tis shame forAnd all that thou wast?","title":"logs"},{"location":"gpt/#king-r","text":"DUKE VINCENTIO: Romeo. Provost: It do trust me. ANGELO: Our lord, I cannot speak the good word: Since I havewas not it, I do'll therefore know Will a divineade me speak. ISABELLA: O, let me hear what you did deliver, By 'twere pity to him. This friends, be his head it, with one that's no matter, look to me as this lure of the prince a curse; more stands on't; No, sir; not a very weak of mine, If not that. Now, andso death's like death! I see, and hear Montague, my Montague. Who knows thou nothing; be not gentle, I may not, yet they shall be so, then; And had he come'd withAnd the oracle: I'll have your heart: that I shall follow that send him; that, we will warrant you, andPEY? ELBOW: Nay, he shall get a man to me. ELBOW: Look, very well, he shallEL' the better royal, which I do do't But I shall find, a time will be from his spirit. I prithee, take one: give me not your comfortier: I will not be come to-morrow; let's entreaty? ROMEO: I beseech your grace, sir,-- BISHOP OF CARke, my dTwere 't. BUCKINGHAM: I will, my lord: nay, You will have you go with me to please again. BUCKINGHAM: You will not hear duty, but for I will, sir, You have not the lady will with thee love But what you may! You are sure of he is to be a doth valiant; I will wot the prince: and have first to-morrow then. ANGELO: Thy business is the more and that my brother's sister. ISABELLA: It cannot be so. DUKE VINCENTIO: I say you, well; for if his present is the WARWICK: And thou wert enam'd; I'll not be mine own? KING LEWIS XI: What man is none than that at his house, That bear their down heart to Edward's-t thou him? WARWICK: Bid me, noble lord, our friends are fled, And sit to be true; and with all our sonsiss, From forth the most defend in theile departedath Be not born to make a husband as if I send it to thought that I did weep, And quench blood or not well. KING RICHARD II: O I was Clarence! What'sFL words were it: That I, as we are, that they have no friend, Even in the blood of heaven of fight, Thou p country's blood and'd. Come, and go with thee go; And, as mine I remember as any man's son As merry as I told the people and am mighty As'd an hour begin and bears our hands, But shame no other Paris and her be drawn. ROMEO: Is my long come. carry, let'st thou father In this remembrance of love: thou eat not 'em most noble swear tongue for what thou mayst, not wert between. The Earl of heaven of Warwick's love, this man, That's hearts, the true king, myMON and Sir. And unlook'd in the princess slain, Is beg of hen, and did yield in their arms At no moreOR create give from their hands, For the dead George of Clarence to his king, Who spake me his enemies shall hold it. BUCKINGHAM: My lord, my gracious lord, You had a power of wisdom cam thou out of mine? KING RICHARD II: He hath enem this, people, but I'll win me. CATESBY: Then here, my lord. KING RICHARD III: Nor I, the crown that queen is slain, To take the devil of our other linean ground, Not his our kingly curse people's-Which they that Is put to have open honesty Either to come in their arms. QUEEN MARGARET: ANTIGONUS: Hear you not? First Citizen: Come, sir; go, be it must none but you: I am ta'en and leave of you; and, then, go with me. First Servingman: Why, we will, sir. Second Servingman: I would not, no more: the provost. Third Servingman: What's the matter? Third Servingman: What's the matter? Second Servingman: No, my master, I can tell; I know how it you, thereof he m it. Clown: He's here; we should be a witness to the purpose. Third Citizen: He hath done what you are done. Third Servingman: A Romeo, sir, for them he would have found baw Rome of the world. CORIOLANUS: But I love the child, Not that thou hast other of thy life. Third Servingman: If thou she were thrice a man, And had these griefing force royal royal queen; And I am son, and I love thee myself. KING RICHARD III: But I will be poor; which he is he made? BUCKINGHAM: My gracious lord, let's see your grace be company. KING RICHARD III: Why, what a fellow should be? BUCKINGHAM: My lord, 'tis a subject, proud Which else, which, had some him, it will with heaven Till wind manage our doth made over his land. Why, proud I have ta'en the service of the people, That would not have more better that, Which will be satisfied; and, how his ignorant shouldER' He seems not? and I have had rather thy, To do my country, to make the'd right, With all the envy of this loyal, Our holy and be England and Duke of York, Not in his second soul I and have been The which never bid us, and call King Richard's lord. HENRY BOLINGBROKE: Welcome, uncle; we shall not be unto him; But so, my good son, I I do further. KING RICHARD III: Why, then, I confess thee, lord, If thou hadst never yet thou yet hadst never, But yet thou wilt wert up in thy horse. CLIFFORD: Clarence and Gloucester, I will not bethink, But that's not yet did nothing: but the gods The times of revenge! RICHARD: Nay, bring me what ancient, and this hand.' KING HENRY VI: But, to thee speak commonth very day! CLIFFORD: That thou wert so disgraced me to me. KING HENRY VI: Woth he the oath that I should tread upon our way? CLIFFORD: What may your grace in poor Henry's life and him? WARWICK: No, like a bawd, you not so; and, and for the poor one did, he is even to give his l. WARWICK: Uncle, so: I do intend to him in all, I hope. KING HENRY VI: How far I mean? if this be so bold with! The which, his queen, the queen, his love; And so, gentle king, may they do you in at night And fly him in the slander of his king, To be revenged on him that, so should you have; For, by this way be you found you to, She's a woman. ANGELO: He will not see you, sir. ESCALUS: I am aTo your request in hand, you shall in arms You must give, and yet go by, a course it is thence. ESCALUS: LARTI not mock mine own good. ANGELO: Do you hear. The duke hath forgot you for what you are? ESCALUS: BUS: for whose offence? CORIOLANUS: What to you? CORIOLANUS: O, worthy madam, And have I too most, you shall not. AUFIDIUS: Worthy sir, farewell.","title":"KING R"},{"location":"gpt/#vol","text":"Had not himself, nor known she! Provost: An me, both that; he did, sir, to steal-on Her mother's Romeo! O old faithful friar! JULIET: Why do I more than that I should never speak of. ROMEO: Nay, rather give me leave me that I will; For I shall find some better happy days Than lay, a man. FRIAR LAURENCE: But slain, her hath the next day a little. ROMEO: Nay, good woman; my turn in thy sake art full nights here. Who is dead and young prince. Ah, how, dost thou find me to thy breast, Thine hast he, and more, a noble fawd. Sir, thou canst not speak, I wouldst thou wert For sleep the mark of thy deadly years? Ah, keep'st, asLord men, thou hast got By so thy voice: there's no power, it is none honest. MERCUTIO: Tybalt, Romeo, whom I have the best I think how I have done. ROMEO: Thy life art thou, that thou art too fair! BENVOLIO: Mehe, and go not in 't. MERCUTIO: Nay, I'll bring some noise for a for that. CLAUDIO: Your face? aarer is a traitor's head to pluck him to the king the house. But which you can do I have to the hand; for it will be, as I am and a block withFor a man that Claudio hath married, If be a uncle. DUKE VINCENTIO: He did know more. You are most little more, he hath done it well, our the city and do them to make me theD:' Aufidius come to me and brother; and she isome with those that hath been, 'twere pity us all. Then, soft, wot each that am aly; For now I did well know. My mother! LEONTES: What's well man? CAMILLO: KING RICHARD III: Be not so? HENRY BOLINGBROKE: So that of God he would have the king his! BUCKINGHAM: I'll make more than my gracious case: I will bear the deed. KING RICHARD III: tis you by king, I would am I bethink, To make an amissant in love That thou wouldst protest to keep post toOR'n; And thou he is not nowSo. LEONTES: Unman, it is; Nay, good my good lord, be satisfied still. LEONTES: A most business: Let me have been since I was to be mercy; Whiles the lark-f orth-she, He cannot, Marcius worthyKE: 'tis well stock no poison. The manner of the envy he spake to Marcius, That with the power is worth dead, the doth be brief, In worthy Romeo hastous tune up. KING RICHARD II: Whom was it so, my child is set on death. HENRY BOLINGBROKE: ClOR, my lord; With late that cons news, for, to that I would Trueass the great number pardon'd, from your head already're his mind i' the there, I will. DUKE VINCENTIO: You do but see your highness of your arms Would know the king's mouth with peace. MARIANA: I mean, a son, it is not a word through a tear for Claudio, and be banished; But when you bid this manner bless us, In whom we have had been wont to do, 'twas upon the business, you I shall call'd his friends to him, and then. DUKE OF Lord: He's tearsark; for long my gentleness' I let me hear; good cousin, adieu! Belike his hands I in his remedy; And I have died to face, my love to me, And I will make him leave of can say, I had rather keep their words with him to keep him Whither, to thy fortune but put off; And see that this terror may be so, So if that cannot: so that's the friend, That will be deservedly of your honour's life, Let thy rage here cut in his grave; One side do not. MENENIUS: Hath the people's great state by power, That they shall have set up a part; But, as minetis the nothing of a man, That is the deputy,tis well for fault, And breathed his honour with the fear of death, That all for that as you are, and we may live. BALTHASAR: A pretty fardman! RICHMOND: Bid him be here? and I'll make me pardon up. RIVERS: What, hast he not? BUCKINGHAM: My lord, my gracious lord. KING RICHARD III: As I remember; against thee, thou wert not so much done, That thou canst swear and thee, for thou hast no cause. RATCLIFF: I will not be not, my lord. KING RICHARD III: Ay, O my children, my eyes! BUCKINGHAM: No, mighty lord, I hope, shall understand it. KING RICHARD III: I know no? but himself he is done; The side must be seen and make me wrong. QUEEN ELIZABETH: That thou mean I sent to see thy life To have no more fit than it is in this world To look upon thy brother's make me hear, The Duke of Norfolk; if thou darest with Thy Lord Northumberland, rouse'd and thy crown, For thou shalt no life but by thy kingdom. KING EDWARD IV: But thou, in my turn, my grave jest thou, You, the newats, and have the ears, Where fruit men have done-bended, then my daughter Of happyness! O my woman! If thou didst, thou tis'st fair a piteous, which he was, And know not 't prove a thousand right while you That you are to have something: so, my kingdom, 'Tis just that I iter 506998: loss 3.5516, time 71.91ms iter 506999: loss 3.6803, time 72.24ms iter 507000: loss 3.3026, time 72.24ms iter 507001: loss 3.4185, time 71.62ms iter 507002: loss 3.6271, time 73.12ms iter 507003: loss 3.5836, time 71.73ms iter 507004: loss 3.1981, time 75.92ms iter 507005: loss 3.6141, time 71.98ms iter 507006: loss 3.2294, time 73.99ms number of parameters: 3.42M No meta.pkl found, assuming GPT-2 encodings... The state of heaven with him, and that his body is humray'd this. WARWICK: And I. DUCHESS OF YORK: My dear lord, DUCHESS OF YORK: HENRY BOLINGBROKE: As far go we as any we toOL? HORTENSIO: I think she is well: then will you have so: I would we were not been then in war's good. DUKE VINCENTIO: We shall not be still be gone and been to see. GREMIO: And your good lord, I would all and like you; and in your hope. ISABELLA: I can know the truth, I am your father's house. DUKE VINCENTIO: It is no gone, sir? LUCIO: I think I would, sir, I know say, I am a friendlike by him: it is the very much of your blood; if Would I be well show'd of you in the bosom And our new-a. Provost: MAMILLIUS: My lord, I'll warrant thee a holy heart: which's with mine, I am too good to him. LEONTES: I am none; Sir, for your son I have here in this business To the justice of your father's worth. PAULINA: It is my gr in that I were, To the most of your own: your therefore have Be consul: Come, come, go, go, be much here too much: I am no more. LEONTES: They are such a and the and yet. HERMIONE: The better, sir; I am an't that you have notity. LEONTES: LEONTES: Well, well, what! LARTIUS: I am so. PAULINA: I pray any I, sir? PAULINA: Alas, poor woman! What ha HENRYHAM: Too true Clarence was a holy bed.","title":"VOL"},{"location":"gpt/#camillo","text":"I shall have had a brother of your son, To yield your brother's brother. LordOUCESTER: CLARENCE: KING EDWARD IV: KING EDWARD IV: Why, then, think you what: go-morrow, son, That you must not do that I am just, For when you can, my lord, my brother, Henry With your brother, if he bring end to your last. And, high-ed, if you see her you do, He would not have been thus an whose speak With that most un Butice: I would, he were born, a name: The house of him I'll speak to see your wife's. DUKE VINCENTIO: 'Tis not her in, sir, his friends, which Of wrongstAName to the earth, with a little way. ISABELLA: I am so mither'd by the point of it That I have had. DUKE VINCENTIO: O, let me think; and let me see a word. Provost: I wouldISABELLA: Wilt thou not, then, not in a very kind for a very for me that I did use me from him to speak. And I beseech your majesty and your bosom And my great opinion, have a wife and father, Unto my brother father and my well. FRIAR LAURENCE: And there is my hNor talk'd be no sworn to death But that I should think 'twack the part of you In the duke's daughter. Nurse: A love a man that his blood is more a life Than this may standow make her false friends. Nay, more farewell; I not for't; Our unf shors'll ask you: come by the man-likeent of this lady is the least, But to the everatter of me dearest day, Or me but one earth, like a thing that Not so much sorrow's, as many more is true Than this in his own! as our bthe light As I do! What's in all my shame already may in openings! You, my sweet HASTINGS: Be early what would you say to our lord? YORK: It is not so. GLOUCESTER: It is. LADY ANNE: sweet my old grace, that thou follow'st! GLOUCESTER: And, for God's sake, till you are three years. LADY ANNE: What, prince? GLOUCESTER: No noble thing I should be a happy king? GLOUCESTER: Away with him! DUCHESS OF YORK: My lord, I must not need: Yere to my children's love's loss of thee, But thou, even to the king, thou thy doubt'st tongue. KING RICHARD III: What is't o'So? DUCHESS OF YORK: I, by my life, I have done, but so in once For in the virtue of our blood: without that kind Was never so far gGLOUCESTR well, And I will follow him. DUKE VINCENTIO: M King Richard, give me leave to my thou thou go Be at home of these good fairWARD. But of what, thou dost to do Edward! B vouch'd: My lords, more than you shall please our royal name. LORD ROMASET: So, to save the king; yet he's just, And come, to answer me in him my country, And you shall have you where we did say the prince? DUCHESS OF YORK: My Lord, may you do this dangerous lady, My father's heart, which your hand did thee yet be The first one of this noble Edward's king; And yet that is no more: come to thee, For I am we of my your love. KATHARINA: I have a very. KING RICHARD III: sadful lord, and his part-f ready I come To make the king a king from him there is That he's made good. Come, let you go; But there be 'twere noIC at you all; But you shall have France, for honour; if you were now have My body hath done to banish yourself, To prove a good will-dew heart, Wherein my noble prince was comfort To make a man's name and a great bed: 'Tis well aKINGBe, he's in a child I'll be by my husband; let me she: I'll have it if I use my sake. Why, how, let is in that sword this traitor! HENRY BOLINGBROKE: Richard, help! but more than his that I'll make my tongue to love thee such life As thou art to Romeo. KING EDWARD IV: This? but doth he not Edward's death's death! HEN MARGARET: It is more than no more yet than by it. BUCKINGHAM: What, is it so? GLOUCESTER: I shall be there was in my heart With gracious lord and my good lordle brought He is well: but I hear you will Give me to my duty, to the world. First Murderer: Because thou canst love, that ever, thy love, thy child, Thy man shall take a grave on thy wretcheding. Fare not thy sword, content Camillo, give Of your she on a my master and son; And here, I beseech your highness,-- FRIAR LAURENCE: eyeTYou may my kindred again! ROMEO: A good things still, this is this day's run Doth never speak,--ces nay, For I have seen thee very like a fard blood, This were thy poor, which bestELLIO: Let the more be put, sir, play in, I'll tell you is Edward in such deep men! KING EDWARD IV: By heaven, I hear no more: my brother's heart. HEN ELIZABETH: I am come to have a more worth in this. KING RICHARD III: Allak thou, God! it is not my son. QUEEN ELIZABETH: There is no more I swear than they shall grief is so. WARWICK: And thou and that Romeo's and thy son's face, That thou shalt thousandame in thy rest, Whose father's heart, of grace and heart's blood is come, And with the tongue that did not be there. BENVOLIO: T Gm would cause, I wouldsthip wish it true. ROMEO: Is thy life- vouch's own word. MERCUTIO: My lord, I'll take thy choice to go Hath yet with an heart-day's death? BENVOLIO: I dare not for the best upon thyPle I talk of thy Ver lives. Come, let's not A noble man: For she hath yet a man to fall? ROMEO: Thou art not so well:OM pity her: But he's an but one. FRIAR LAURENCE: WithSecond, sir, a thousand duke's of idle hand; But I am sworn to it I should not stay, To be a life as thy honour's love: But let me rest? JULIET: This will be, to put forth; there is no man. DUKE OF YORK: Well, me for mine; let me by till thy husband I have no brother, till I am not a king's son: For I have done, I here hear you speak: By heaven, I am a beg by my life, So far as most in a man that's the mour of! O, that I would be less! the great which o' the honour, And that I had; but then, I am a-beLEath slain, Were kill my takest! If never, run Aufidius, 'Tis thought it would you are at enmity That you did but so much is, and And I'll lay down out whose't be fullle In such most I have kill'd his have been such his, As so in the world is dead. Messenger: The man that he is our lady. KING RICHARD III: He kill, my lord; who, we could not say 'ld have so much. Come, come away. Of God's name and our fair? 'Tis oath? TYRSON: He's lord, my daughter; but I doubt not more, Farewell; and 'twas it be before I did Or not a one that she is not to be thy tNor in a thought of the world was not enough to be the king. Provost: DUKE VINCENTIO: How? DUKE VINCENTIO: O, let me go my lord, I would youheeio her, prince, of you, And make me be seen to the people. DUKE VINCENTIO: I do believe it, to grant and love me My state English yourself, being full of your love! O prince! one of you a man that I must not all be the king, which is mine and him. LUCIO: But, madam, an'tess'd by the do break your honour. DUKE OF YORK: I, for an Murd-- KING RICHARD III: What! what goes this? at' lie I'force? DUCHESS OF YORK: I would I had a poor day for- thou art made it. DUCHESS OF YORK: I should I will love thee any Isabel, If I have seen the king. Provost: DUKE VINCENTIO: Good Romeo, Romeo: 'twas done, As that in any man have done to you wretch with him: So, well I desire it, to my heart Is this the day of Edward's wife: Thou most Richard, I fear me, Thou wilt make thee as thou wilt be a thousand As I am, as most gentle for ourselves: To this my husband's fair day, my soul, I therefore let her stay with heavy kiss: And, as I could, but I would, my good lords, Ere you be king in oath, and my death. LUCIO: Why, how thou hast done this man with thy love! Farewell! thou dead too, what 'twas no time Thy other blood should come; and with that WhosePOL know to fain: 'Tis so false, to beg mercy, their pluck'dine side For their goodILL'er's hands. POLIXENES: What, be there enough' the king with the king: Clarence other m O Warwick! if it be done, And that my good cousin, I wot that I know. LEONTES: How can you think! We must be this be you; this power are My true son, my brother. AUTOLYCUS: Here's he that did keep her to be: he's a par and a the duke; he is very well said, he doth: He had not so, I do fear. AUTOLYCUS: I may say, sir, my queen, dear love. Clown: My lord, you will, by my life, a word; the he duke's daughter's wife! MAMILLIUS: I have not been i' the lie. DORCAS: I know thy canre so. POLIXENES: O my lord! MAMILLIUS: No, my good Paulina,-- DORCaius out,-- DORCLEY: So much to it, sir, that it may be received To the sons; your pleasure where, the king's heir, Have he been nothing to be depose, But the poor law so far the rest is dead. KING RICHARD III: Ay, if I have Clarence's bosom forth. QUEEN ELIZABETH: WhyWARD, Well, pray, go with us? he is very comfort. RIVERS: No, as the devil'st I am no more more; But I shall have heaven with my life to go. QUEEN ELIZABETH: How long the justice of the earth is not mine? GRUMIO: prison does: Come, sir, it is no more. And the rHAM: My flum, will tell your Lordhip; here is mine ToWARD; and yet I then lay at arms.","title":"CAMILLO"},{"location":"gpt/#york","text":"God save your lordship to his own Lord Hastings! I but on such right and said 'tis well. KING HENRY VI: O, then, I'll away thy life to bear. QUEEN ELIZABETH: Come, come, come, let no longer, call to our daughter: For now the news is come. KING RICHARD III: O, but the father that is the king, The love for maid truth: 'tis in the needful Richard' death? QUEEN ELIZABETH: Nay me, gone, go with me, in good I'ight in thyAnd shall the sound wrong. KING RICHARD III: So much theseford still he is: His name is troth thee well my death: What, that is not so much, any of thee and let me think she hath no further As, nor any soldier, nor pardon, But that was she hath done, she should be. STANLEY: It is no more of your bed hath best not true. KING RICHARD III: Hath there been, and die out, so much to me The word of a love: what art thou? Second Lord: Thy father lives, that therefore can not be I take theUTA way out of our offer, And, as I say, for we shall not stay. LADY CAPULET: O my hand, a fault will be more never heard Or else moreare. CAPULET: Under this boy, she is of the Duke of York's son He live from mine honour and his blood'st To take us good; and, with much subject Like to the cause to be thus. LEONTES: You have content'd your most mostday To speak your ownness and my soul. LEONTES: Why, the Lord Angelo, So York, as it be father, for his life, To call them hope 'twere yourself and dangerous. CAMILLO: Why, what a thing? LEONTES: Was everBut in such a time of her, Or in a good will of this unisest? KING EDWARD IV: What, lords? WARWICK: And Warwick Clarence at the eyes of justice! KING EDWARD IV: Why,ere no more. CLARENCE: GLOUCESTER: CLARENCE: HASTINGS: KING RICHARD III: Death, brother, are you through my soul Sray to me, and leave me to my uncle? KING HENRY VI: This is thy brother's son in want these hence, Lest as I say, his Earl of Warwick, And tell himself to the Lord king to him, And he had been the will This comes. QUEEN MARGARET: Why thy words have there the blood for this land In that thy heart that should have. You, God's right I should have needful nothing set upon, Theself I am: and yet thy doth-- QEN ELIZABETH: O nothing, that my lord Richard will have. KING RICHARDISET: Yet, as I come! KING HENRY VI: I know the name do march. QUEEN ELIZABETH: O, I wilt thou not know that sorrow hath To hold the world with thee, and make me think How I am of thine. KING RICHARD III: In whose from all the shame of save, ThatCORIOLANUS: Why, how's there,-- VOLUMNIA: Let's hear your heart? MENENIUS: much for my very word, I'--Come, let's go: I am please her I shall do it, sir. MENENIUS: It is a cause, Pray you, get your words. BRUTUS: What, our good lord? SICINIUS: He shall, you shall be; he are three gods Is full voice. First Senator: MENENIUS: Ay, but I will go. First Senator: I know your Rome. BRUTUS: Is the news prove true, my lord, and thySo, as I And see that this beable and mostadman: Hath not a high Mercutio? That's more to say 'twere well;' which I pray I am too more than some, but so good For one to be done. LEONTES: We know my friend, I am theWhat I of your body, if But who youath done the noble duke, But what is done. LEONTES: Good Marcius, you have lost thy devil, to lie all Good night. POLIUS: The people did. SICINIUS: Are you all Aufidius? First Senator: You would he were dead, but not so much I: But now I'll call for it. MENENIUS: Well, well, no more. CORIOLANUS: As I had forgot the world, you have not a kind of love Must be a child to be satisfied, For you being no man to her death. BRUTUS: Away! the arm Of your fair wife'st, For she was my enemy. What doth the man! DUKE VINCENTIO: But makes him, good father, Thy poor prisoner is your time, I'll have a brother of a son too; The side are all betwixt me and his brother. And I have need not; and, as I am not The goodly gentleman is too, do not But else 'INCENTIO: O,en O, good me, indeed and I'll thank thee not no He hath deservedUKEill live to do this; And, if this beNORine no man's death is cold. AUTOLYCUS: Away! he is not made good, sir. DUKE VINCENTIO: My heart, my good lord, did't not't. ANTIGONUS: It is of this; We know of you are now. Servant: My father is at hand, Who came Camillo: 'Tis goodly: I'll be the bosom on the world: When the common misfoler of this battle, To","title":"YORK:"},{"location":"gpt/#iter-599988-loss-28334-time-7135ms","text":"iter 599989: loss 3.3947, time 71.09ms iter 599990: loss 3.6317, time 72.32ms iter 599991: loss 3.5352, time 74.15ms iter 599992: loss 3.2102, time 72.34ms iter 599993: loss 3.5483, time 71.71ms iter 599994: loss 3.5832, time 71.76ms iter 599995: loss 3.4675, time 71.01ms iter 599996: loss 3.3202, time 71.27ms iter 599997: loss 3.7681, time 71.95ms iter 599998: loss 3.6641, time 74.20ms iter 599999: loss 3.8146, time 71.89ms step 600000: train loss 3.6565, val loss 5.9145 saving checkpoint to out iter 600000: loss 3.3908, time 193.50ms number of parameters: 3.42M No meta.pkl found, assuming GPT-2 encodings... The state of men with him that, by his brother's, The very good men, and hisWhen we revenge to Myself, like being now in the airing. GLOUCESTER: This gentleman is set out of other. LADY, lord, give way, and go we to our cannot, And I will leave them to one the YORK: Why, then we may come to you to the: I would thou wouldst hear; war's still he thou done: O, is 'twThy graciousYou were at the heart. First ay: I did not stay? HERMIONE: You will not live But sir, nor little's by the he'll take it down To can. You are not fit to me? LEONTES: What, youPEY? MELMy words may bear't. The one would have so. ESCALUS: I cannot speak in heart to come by with her the thee: never only of your love, O, He has with him most peace; he'setheth once, And therefore do you any thing he would have you; That, in't your eye, well- ELman is so long a Jof which's made youity. If you, you may hear that you are, wtis a shepherd only may he say Bohemia. DUKE VINCENTIO: To your love. LUCIO: ANGELO: I am that king. DUKE VINCENTIO: I would I had been on to the A there: Yet must you be, sir, to all I know so. DUKE VINCENTIO: The duke shall know you say speed again to yourself you. Provost: I will be an't. DUKE VINCENTIO: It is a word of me. ANGELO: TISTRESS OVERDONE: Ah, howio! not all as he being As is a father of a e'er these adTan your brother is born. MAMILLIUS: I am a word to-morrow; youpt, But you might, I shall have had some to him thus. LADY CAPULET: If you were mad? 'Tis full of what you should have. CAPULET: What, say'st that I mean to me to thy name? JULIET: I am the more. Nurse: Is a word. JULIET: What is the news thou bring? ROMEO: How shallest what's a d see of them! ROMEO: O, give mewence, that comes in the time Of this. I am not born a happy day: The which is an ear thou now to make me speak, Nor how in that word, thy majesty worse, To Romeo, not to theage, thy dear queen, I am a comfort that is my wife. You that's a brother, to the duke's e'er, To have his mistress king, to-ake: This is the love I might have doneish the country, That I may live in theWSo name lords that means That ever had been to doway on! What think you, if you were you all, Will not my full very reason for a great bed, Uniansch high the heavens? answer this Of this fair prince's name? CAMILLO: I will, my lord. LEONTES: Pray your father! That's true-Sir, and you are son. POLIXENES: You are rough sir, give me Lord: let him go your seek. LEONTES: My fair lord, Were I but so much, or do I see, my more proud, On this most gracious lord. ANGELO: Thy great youth is the forth knows. ISABELLA: But 't, an confp Henry!' The which you seem not you both, which would well, That I with dearLE RICHENTTER: As now I have nothing: With quoth so, my lord, and all at night We cannot speak, let it to remember me. CLIFFENCE: Is some thought of this, we may be a cause.","title":"iter 599988: loss 2.8334, time 71.35ms"},{"location":"gpt/#king-edward-iv","text":"LADY CAPULET: And to me have me bawd, youngly, I would say, If you say so, it shall know the world thou thyself. LADY ANNE: And, I am that day to last with a thine. JULIET: O God's sake, is come! I'll make a years upon thy husband; And, all am grief of call thee all this good. QUEEN ELIZABETH: It is my father's; it is more A dead, a proud; for if she be king, I may not live another woman, I am not a father that will take it; And now, for my father, by my whose fineoe Provine and my wife's son brave father me: But I hope, my lord, for I was seen, With mine own love to-day; let me lest: So I have done, yet comes one. DUKE OF YORK: Thy kind-morrow spake for the grace-night! DUKE OF YORK: ay Richard, that dothst thou do to love thee? DUKE OF AUMERLE: Then, I am mine own again, or I, The truth of day; and with a kind house Hath made her dead, who have manyre many more That I'll hear him rather and say, for my death, I give a purpose. LUCIO: O living, my lord, it is gone to leave it, And you shall have of this good or'dN At my poor brother's kiss: and I have seen, The one that's made me welcome. DUKE VINCENTIO: This oath have as high as the wisest, then other mure of a chide on: the post of those: You have not there here are comeable. LUCIO: But in this while I should have them; if I cannot, I do put the fool: then, like I come to love thee With a set upon my master, which is his pence of the king; even so in his hands, As I havech you at his best, By that it may France, for honour! if you were now have all; I will be banish'd, do not to prove The corn of honour, and I will not The noble touke: but I will not have any name That I, and think it were, if I could stand, Most might be in this fair way thy hands And me shall the face my husband'sJShould be, To young princely to thee here, be long-s, Take it again in noCOR'er a happy depb not. ROMEO: O Romeo, nurse, how can your dU is a happy In mine honour or anBHAMELLA: He shall have a poor way inch by his honour Gless worthmen; I warrant my counsel was more. DUKE OF YORK: Let me I speak of such, my lord, That very good to be this, before the time Wept with Iixt these words. DUCHESS OF YORK: Why, so I am his wife, to do't, Yet hear my lord; 'tis so; and I, I do him! DUKE OF YORK: Int note was that ever wasR pleasure true: He which's anMENain'd now: heuck me not his bawd than I have content the old gold-bed, he's dead, That may not beoe to be thy face. DUKE OF AUMERLE: Whom is the way, to youQ is it so; For I will not: so she would thou hadst no time Than in me but not: but that--tis not so, For I should, I think, I please you, my sovereign, And by our country's breath shall hear thee, To lose a matter with a word; if we At the most little: to such nor thought it was, Unless I think had been yours, for I crown'd; And, with a word, poor soul, that thy love's will do it. BUCKINGHAM: I know the gods. GLOUCESTER: Pray, be these my bosom: for the time I'll pluck the pack of your house face.","title":"KING EDWARD IV"},{"location":"gpt/#king-henry-vi","text":"WARWICK: And Warwick's death will go and did to us. And yet not be fear'd in all. Go, Cates, and not thy brother and brother, And come, I'll swear the duke a good To be as free as little as, so myTis-- That side that would kill'd my husband, so Whose honourFor it is lost grave in-- Whither dost not be long as long as Henry's love My father, Edward, to my tenderbr own arms: Till your three days give good thine and your eyes. KING RICHARD II: O I wast thou not dead, say I know thy mind, Yet thou that dost in beseech at us, And not to the people that brought you to me And think you of our good rather shall be Be it to me but for my best good. First Senator: O Clifford, O, it is the more! MENENIUS: I would he had a people, But, were you not't; and you, my lord, You am of what you shall be so? BRUTUS: When you are should have bound 'em. MENENIUS: So, most it do. BRUTUS: Where is the love of this fellow, that must not? AUFIDIUS: We do not say: We were to do't. CORIOLANUS: You have been a cause to need. VOLUMNIA: O, indeed, I'll to desire o'er a counsel, And I will play the us. MENENIUS: My noble wife is me as good as well in true, To have an those water's better and isar'd. He pluck'd with him, and you'll have all the news-- And it is as good as Montague hath. Second Citizen: That, my master, which is his friar o'er-s, In that she had done the noble duke's from The noble head to the-night bed-like. First Citizen: Sond to the king: But I hear, madam, let him be heard it Of God's name, that is ill and great blood On thy be these; yet every w thought I with it, And none but that I doubt; and, in your love Is it right: but I, my poor dear cousin, I hope that I had not to this brother, But I will stay in the thing I say. DUKE OF AUMERLE: Why, lords, what aunes may? DUKE OF YORK: A gage, prince wouldst thou tell thee they! Thy brother's death! DUCHESS OF YORK: Unshe's aHath To Romeo's name the traitor; And you, God and I will see thee a and that Of living men thus they had been in this love To the duke. QUEEN: So, they are not to be the name of all. KING HENRY VI: But, madam, be content for thy thank; If thou dost sleep at thee? Now long rest is dead; thou wert not thy f after? QUEEN MARGARY: As thou as now I love'd myself of thee? QUEEN: But I shall fight with him. KING RICHARD II: O God's pity where long the o'er the world-and'd ! within I have done for the hour of thee Whereof thyself be too quick for my death, That enemy, with thy warlike blood, And made the king in arms at gone. DUCHESS OF YORK: So shall I do, to so late, Why, proud-hINCENT are lived told? DUCHESS OF YORK: No? KING EDWARD IV: I know you that is a father, to my? YORK: By fair, I thank thee, love thy face Which thou hast welcome of the fool, thou thyPR; And, now my poor queen willBYADYOP, For in my brother I king have play'd heaven: For Edward's face, being gone to Bolingbroke. TheAU's aH no other tongue to thee; For I have sent not to be of.","title":"KING HENRY VI:"},{"location":"gpt/#duke-v","text":"WhosePOLIXENBERTER: Not that that shall tell thee to my grief, But I will not be gone to thee, and thou a present tongue. 3 KING HENRY VI CLARerer: Take it with the day. BUCKINGHAM: Therefore, my lord, I let not beauteousors. FRIAR LAURENCE: Why, then, I pray thee, call thee bring me to me For this my gentleman's letters and her: 'GoodISTR in't. ROMEO: You must beL blood; And thou wert dead with a for him thou this! Nurse: Go, come, come the nurse; I mustETH: I pray thee I know thou canst wish it not say. JULIET: Though we have less, for ourHave set than me, So full of honour, I am gone to Warwick. ? of ourmen born? Second feel so: this is well as true as you, That you are put up and see as fast as fit As far all-f- HINCDand children to my poor blood To this loving? LUCIO: Doth she, when her father's Claudio hath done. What's he? He is in the name of men, With tears of heaven, and their true-ouon put'st Ourself'st man; and, and so is they all; That thou, like an unf swift use of blood. O, I have thy king, and thy king, That thou art a rest, thou, then now in thy place, Take thy hand in thee to do thee most am dear. JULIET: O do this love, and be an O! O's thy title, Is more more than yet thouThis: now, and that thou art To be it? 'tis not a tas beg man in seen an hour, having whom, he will not tell him. The means of you, my lord, is your lady's son. First Citizen: Away, my lord: we know they do us all. First Citizen: Woe to you? then you have said you were?","title":"DUKE V"},{"location":"gpt/#second","text":"Gentlemen, may she be put for it. I but think it was a bawd, I had to do so; For, in good time, a very well-baww gracious. Nurse: Why, how now, I art gone, Saw-b, ere, I hearon! JULIET: OGLOUCESTER: Here, that would he were so sour to he. LADY ANNE: Why, Warwick, how now in day shall I be! GLOUCESTER: gone, my liege, in what! If it be I had to die their high. KING EDWARD IV: Then am you patient; and therefore is Of all my husband Paris, to-day. GLOUCESTER: O, I would I had rather, but for thy son? KING EDWARD IV: Your death, my good, what! that I am there. GLOUCESTER: No, to thy name; but very it is not, For I have, to go with me; why, he's dead? heir: We shall not, my love, my lord; But, Warwick, do not justice her out; For therefore, so it is his hand that is: And yet our country, we are, nor I. QUEEN ELIZABETH: But what means, good father, be that state? KING RICHARD III: Tell 'p thee, an brow of all. QUEEN ELIZABETH: Good prince, so! KINGain: What do you thought, I hear it, I know you well: Your face I am not in your good counsel: Then be it then? KING RICHARD III: Ay, but he, but he's no more than a country. BUCKINGHAM: Is it none, good father; you hear no more. KING RICHARD III: We have done a better, our person. QUEEN MARGARET: Call so it, lords, have made him our hands: Berely as well as I have; for this, As for ESCALUS: I think much; but they find't their way out, but to die The people, and he hath yet show'd! ISABELLA: What you, I'll givehe outOUCCIUS: I am in need. ANGELO: I was a kind of all. CORIOLANUS: My gracious brother? LUCIO: ouson? ESCALUS: UC.' is there a thing? VOLUMNIA: I would it, sir. COMINIUS: Let me but stay. CORIOLANUS: The gods too word, my good nurse, I come from Rome and of the had the gods alt have been in the presence there that made; The people lives upon our TheABes; So therefore you, I cannot do you, you Were, my good with me, your father, his lady: One of you, he has done, by the honour To Coriolanus. CORIOLANUS: Haply, I mean to us and on. VOLUMNIA: ine, Lest, take up the duke out the traitor's son He was a like a father; who made My best son: I will wish you his love. Wilt thou? What's this? what? will this, so I here, Were in me by my master, I am like. JULIET: Richard had his grace, Nor, since the life that it hath most his, For much more shall be, and in that word. Nurse: JULIET: O nurse, I fear it were night. LADY CAPULET: Pray, you have all this day a may have youak. What, is not so long, you would have been long A noble deputy, which you come toius; And you the people is, you shall find again To see you knows the place. CORIOLANUS: Away! Second Senator: Come, I pray you go to you: you will hear me: Come on, madam.","title":"Second"},{"location":"gpt/#menen","text":"DUKE VINCENTIO: What, if you do not, you should be been well That you shall have; and to the alone of my heart: Do not think that it is hard to take. LUCIO: And in that world where he cannot be; for, I do see You that never, I have heard, but not go; I'll the souls of more and more That is a good way to hand; take TheSIC he my C art, and myself to please. Good ladyENS are thou what an openon 's tooNot, nor what can part in this fair speak. NORTHUMBERLAND: Come, cousin, you standNow down toWhich as these blood As thou wert so now in a comfort's hand. KING RICHARD III: My dear love, my lord, my sovereign liege, MostFrom earth I come for Edward's death; And longum thou, the fault, but not thy lord: I look'd, to be fear'd for me to live. GLOUCESTER: I never will, I'll speak this. First Murderer: How now, like! what's the day? CATESBY: Marry, my lords, shall you seem inh nor wits, Or take you by the once. Second Murderer: O God's brother, thou art mad, and begish'd. Second Murderer: 'Tis never gentleman, but what of my counsel But to my will at morning. Second Murderer: How long as I? Second Murderer: What with a name, was this? First Murderer: Alas'Twixt me, and must be thy daughter: Doth set the world upon thy counsel with thy thee o' the face, thou hast made them aHere. Second Murderer: 'Tis more but not, dost thou havest Lewis, That he hath leave of more unto of this Richard. CLIFFORD: WhoCAP thou wilt fall so of he had been of! RICHARD: And bid me stay. RICHARD: Messenger: Give me thy oath I","title":"MENEN"},{"location":"gpt/#1000","text":"iter 997: loss 5.8675, time 724.04ms iter 998: loss 5.9554, time 722.33ms iter 999: loss 5.8129, time 730.23ms step 1000: train loss 5.9102, val loss 6.0530 iter 1000: loss 5.3077, time 1634.94ms wandb: Waiting for W&B process to finish... (success). wandb: / 0.001 MB of 0.001 MB uploaded (0.000 MB deduped) wandb: Run history: wandb: iter \u2581\u2582\u2582\u2583\u2584\u2585\u2585\u2586\u2587\u2587\u2588 wandb: lr \u2581\u2588\u2588\u2587\u2586\u2585\u2584\u2583\u2582\u2582\u2582 wandb: train/loss \u2588\u2582\u2582\u2582\u2582\u2582\u2582\u2581\u2581\u2581\u2581 wandb: val/loss \u2588\u2582\u2582\u2582\u2582\u2581\u2582\u2581\u2581\u2581\u2581 wandb: wandb: Run summary: wandb: iter 1000 wandb: lr 6e-05 wandb: train/loss 5.91025 wandb: val/loss 6.05303 I my, your the am; By areU he: WAR H That a me, I am father Or: And to her I I, I ROM, lord:ANG is IET. ITER, He Hold can for my your am, And if yet be'll to the the a life the the he a my; P: Sh, have this, my, A,, A thee- I a tell: O. A be any, Your you, I my death For I, this. You, my As I and the PET's artUS, My life than. Have see! and hand to God of stand, With, a many me? Where of have think, it to I have not, That that that pray, As the my I not thy the she lord shall a thyIO and would poor, for the: and: What and he, ' so it, did a will his king, a is Th: Butour Will, Were,,. She, COR the her as, Good When. For lords. and butET is there your me? and shall king the this. C, thy mine. O The shallUS, That: GL: That, That, Then, here the But, What: H the, D and, NOR toEN! if must, The the,US? To my the's poor. Why's will have he'll can how man Some in cannot than come, Even a in, him. Is you anyENT me; so say, be sir, the the Itis the, be'll man I them. What. So thisAB he, That that the will, That the'll:I a king, That ' Would Now, have my the a: To kingUM, I'll b the must a do thou, H.","title":"1000"},{"location":"gpt/#were-his-thy-the-the-the-ill-or-all-cannot-i","text":"To No for they not such time, name? And was have to not me, Come of must know,, I be the yourUS his such I be my. How, For me the have have: ROMING To to when to, My: as And men, An a B all or And thou is, To man, And the: Come, am sir, An, And: Lord queen for, they my, that he part This! It in my trueG do in: Is? And we most, that the BKE: Is an NES? From say; you and, to's knowING your I his But V for it: I lordUL from well, Go in me, it the be I was. but: the her as IU the he our life; if his I most in I the is, I But in hast: Thus and your noble the do take she be: ' I'll a I your d ever, be not allEN COL to aOL, and not he, you is I make, This you not? I And, beELL what there: Do her thy here, not not Not crown your in a that, if death would we you not the w made of his me and butI not, What, H it P can that The the, So: COR H This: if'll B, butET with will hand, I the a N to must a, If, Which I would your- have by the am isIO, What no come is This aIO: MEN been d: WAR in a not, '; name: To sir, I most NotUE men, I What and d me:. my? The's have a us he the I lordose, You not did, Here, do an a she, my we a he: BR thy Why she. P love,, COR this the that how, Of what out H thy: COR his me you, And as NOR you as and that But I Yet. I no crown, say: As have: and wereTER:To: KING KING. ROM be you my the yet, some one? Second their a theI, the father this it not the his. A as not: ' it thou I theICH. O a lord.I it. That theES to he So. But! That not do. I: And your Q. I What IIO, is lords, he the stay that my come: How it were. H not, D. And, that be'sEN: H P: And my that have that not't to they the thou thy were If in; and the for know N In: for my the- never do tell R d it, Not, First, of I Come I be be hath a be her these me by this: and the it goodUT As My when, MER must. H And have the you- sir: and your To be aT, which is; That one shall must you the a, That make I that No'd. This; If now, good dIO most you time, My. N you, and this is have will am, ROM of were shall lord toUS, ROM is be his him: Sh: and her their: H to beEN I the name, Prov not all the you thee, The it, that was,IC How, As such not no IINC, our I and this is not, C. By but I W will as his Sh! you, To him: and say; she. Sir at notEN. Your me, for what is, Q them: Is Sir, one a him NOR with ROM mother that be that see in, H. I D, the so,, This man to She come? D Who, but; by He, H, I I I should my sir. As, take my as or. Let: Is Where his he, No his you be D have,And you He that in man thatIO And You in blood should That: As was lady. Now with: First, theICHEO isI very: Ah, will an- I blood him a, house, that part, my: MER, so, And I but you: What: Your come, Q, To, Now, will he see: Be, His to his an shall then,-- I. I most, your the this By will'll and For a the a a would he At I'll? I: Un my head; They thou am, I. It: To not, my noble in so, and O. By art? Let, they is have thou: COR; and I The But Th ' An are letOL my. And My will, What me, A, A good Second. Then: Sir:To am art D. I the thetis, we one. I Lord Th. D, let, Now, MEN, the, To which And. TheUS so that an be give, be so, Come then him. That can the:Y with by; you give to'll A this to be the letsIO to be the And have'll, These is a thy! I And: OfEN cannot'llUE with he my do The did:","title":"Were, his thy the the the I'll, or a'll cannot, I"},{"location":"gpt/#of-thou-say-so","text":"be, best The our What: In can should of a he I To, Which J: I: He of his sir- am, If sweet the Now. ROM To, their. men? D, would king: And And one And'll himly name. COM, sir with, And: N am's that you of there thisOL heart: You if son But: WereET: if I never, Your in your not? goodIO, or: How Give not in; Come yourEN have most: H:So the I do, If my my I And the good I this or love in think: My Come's and some not I call, to mine: KING you '. Of To But life, be, B you, you, And go: Good theUS and love W forTER to'll II them:But'll an. If ' will, and's part be bear here, A: We you How, Make thisEO's his would love, a when not: N him it, are you, a thou then, How. Cl a a for your! be the part, And take the But, sir, That: Ay, well! This, Second all-: If a I the'll all withUS: Totis death; sir of: And his her thee to this'd: Un, see and a KING, The the be; thy man, some do: And thou ' the the some thy To the this As. That--IO, The I the aKE thou in give! It shall some thou have I I I you the a To I d not am the thyKE a my the this he: Would my life,,, Well is thy, Which: Which a well. There R, the a B so: and UE MEN not in I tell withI if If in the R AsUS. but I the me when you, Good And him. b poor, One MER: To all: and is I makeant upon it I we a blood I. You Is, the, myIOL would First R, a he, Your queen: no people? TY: I. Is shalltw: And 'ES youtis Which his my your my art we haveUE me S his but man, which come: That H the your the, b, His the a a my king? And Go to his Second hath I his more man up, That, What. And do'll a a As He in he have, I my willIO: And, That he I isU You, But. you? Which to a the a, To You a, have the my him, a these this and a him: R most love: that no do make if her For I know, my grace. AEN the am your your the I lord? Who, I the with But: I, a Ns to have so such: KING: J a I day is D crown'd, That But the the. Against when it J. MEN a the. If It love Give theKEant, E, go. Sh: I ' as I ownUEAN: Thus will myIO of And, Of queen: IUS of my lord! it, He would thou? His, And that mother; And N I have, for his if shall, Now, to the am, there that great must betis he D this. C the do, To an, As a them,: ROM, Third, First: For the see,","title":"Of thou say so;"},{"location":"gpt/#i-the-a-a","text":", Now I was,'s that I it you the done Th with do my, That that it: O: Your, our'll And the that ' him like but, a honouron. I dead to her some I: He, If: KING: R thyUS. And Too: For a be theELL, our, And many the but am this, Why, They that the this make, I: Is have am her my my he all,, when the will the to be, This, As may I my their have the To him: H know is say YU ETER, Let. Now. A give is of shall thou, There you, P, I poor, a the some have the your are the for do for you this to your a for or good do, or noble: But And shall he be And good them with Cl's The are a do you. That notown, Sh, I be do it,U O you I be die, Her, by I; she, To be to that sweet all, We be: D more. For the WAR: a the-'s she, that And a thy what, And It, and these. The thetw's know. Like of makeIO. A But I speak: with and: not, PR: Come we If his,EN hear: To I I, N; Of, I, First so the head: To that hisI be you the: That you. You? ' shall I be As He is KING, come KING, the do the the So, With, And shalloth such me,, you by: That to her for I like. Our, As in this love:","title":"I the a a,"},{"location":"gpt/#000","text":"iter 4990: loss 4.7843, time 730.76ms iter 4991: loss 4.9219, time 722.85ms iter 4992: loss 4.6760, time 731.14ms iter 4993: loss 4.8503, time 699.10ms iter 4994: loss 4.7151, time 721.49ms iter 4995: loss 4.7852, time 687.31ms iter 4996: loss 4.5192, time 732.55ms iter 4997: loss 5.0381, time 727.84ms iter 4998: loss 4.6365, time 694.68ms iter 4999: loss 4.8973, time 691.50ms step 5000: train loss 4.7097, val loss 5.5308 iter 5000: loss 4.3929, time 1421.90ms wandb: Waiting for W&B process to finish... (success). wandb: - 0.001 MB of 0.001 MB uploaded (0.000 MB deduped) wandb: Run history: wandb: iter \u2581\u2582\u2582\u2583\u2584\u2585\u2585\u2586\u2587\u2587\u2588 wandb: lr \u2581\u2588\u2588\u2587\u2586\u2585\u2584\u2583\u2582\u2582\u2582 wandb: train/loss \u2588\u2582\u2582\u2582\u2582\u2581\u2581\u2581\u2581\u2581\u2581 wandb: val/loss \u2588\u2582\u2582\u2582\u2581\u2582\u2581\u2581\u2581\u2581\u2581 wandb: wandb: Run summary: wandb: iter 5000 wandb: lr 6e-05 wandb: train/loss 4.70968 wandb: val/loss 5.53077 number of parameters: 81.52M No meta.pkl found, assuming GPT-2 encodings... Not that; You. And it, I to the mother on my son.', PR: And the suour of, Quke, That, not is this earth And not, Now more years will the part: The hands; Not, My son, To I do my queen. TellY: Darest by strength and. And, As Provife's side, And I not the gOM look, which: AUTUS: May not men to dU and you, So, LUEut's lie on a would I can! The suilman, The noble lord up I I do nothing in your end, DUCatter thou have my r LA, For, with to the earth to your wife, Not my or not his hands With you do a friends, His loss, Should have not, as Ibians My mostt to I lord, the world, take you not that you the cause, IOL OFCES: To you, if her my father, I can be your Tower To do my name. That the hands to me. Icius, LE: Why, LUCIO: LELCUTIO: To thy lord, to my day with me? And you more for, For never my power, Her heart; DUE: Good great noble father, this In to an father? And be do for the friends! The lord ClOL OF YORK: And he will he of my hour. And if Will some thing at the lord Bove's father have her; Darep the counsel. As will be those the one, go; come How me: Madunes! By me. And a curse of my. ' I hast my lord Not, And, my lord; He. O Do you have be With blood, R VI: Tell your foe is a king-eseous death, Sir a most mind. That: Who I have ProvUoWARD But, O And in a go, From sweet high is joy upon my, FirstICan had this? WARICHCES V Lay: Come about a people, IUCUS: But this land'd's father, And Could To my will; let my ancient lord Unless my lord, but That, And I shall be I, farewell, if I have said their head, And yourly the time All it which thou have, Which not will; an eye! In me of her not. How I would I DUCost it have the lord, to you is my d'll my common d understand What will not.The business! Glouace; and you, Why your king is your great And Mess not's in heaven on me Say It thy day.IUS: And FirstIC VU, Be that I? AndtwEN Which Best you no is to thou should I do not, Be thy life; And and If you. And I not. A: And he callWARD: To their most we do yours, Dost; Who the first: To be't? A: To my king but Once will But. Thou hast our loving dADWARD IV: The Duke; Sself is the father, and be make you Why, Blbroksarry; From a Lord of that she? As this, Which I have but I spoke, since was the lords, Well, And your course! Sir. Do this grace they were in heaven NEENIUS: Look's, BowY: With in the word. L: But I wost to not the heart To all thee, LU' the my heart, Hre the heavens. And he, we art shall be be your other the last in this quarrel. B me to my royal prince, My life, it may bear hist; SecondUS: But in your, good day, And you, that, And it; it, to give him, From true may be not AB, Look to you will My, In thee-OM do the less aself, Before the, For I not better 'er she not you here: And be a heavy un Senator: MENall him It, It be my suit KINGELBY: Yes! Is my father, To the best, a grace. YATHolding, but such me I cause your sovereign! KINGABILLES: Nor what shall do a ear of the word's tribricMour him PER be. A: Sweet sister, My self, That the grave, The lie' so thy case in put, And my wrong, MENre father. Your child, I do be our will Nor your That KINGOU RABit's with a sovereign's heart as, and been But, good lOW: From your court will you me they friends withESSyly I-US: On the mind, Iie, And if That the good thing the very poorICIN With this is him. POLam'd me Say, And thou I do be be all my word, be my noble day, a Tower to you, Or, That them, sir By my sister; HILLES: Or than was this, Or it, LUS: For none. Have all your dINGS: HULEfore. Of her, Who Th OFABall Burse: IUS: Dcius. But time, when I will be you it the lie me, So me, So my lords, Be but bear all more His better I have thee to not by thy dU me's more may, And he. Go, That the words, which I besinks would he'd in hisENTumerine! Had I will, There has, but when Stand with him, FirstUS: Why: Our father, in CO; Like you it me's lord, But,' he heaven? Be you may the life. Doth my lord, but was you is dead, AETH And see your words. Ander thy will all down, Or in my suit, And at this f Murdaced not, To; LUS: From all the very and his And I must it shall am I am you you, Which, My friend with Inay again: For they traitor; For you but an country? To had the LU, here, had That thou, do the heart, IUS: A: IUartarry. But I but MEN: FirstUS: And they not, LE OFELA Marshal: Are not to heard theICINNow, sir, RABELLARRIUman. Well.And he a eye, and.And show to I be your death have this lords me that, in the king And this a times from I would say, then, your father, FUCUS: R YORK: Now. To see with my queen, GiveBERIO: for your sister, For it. Of your land! And For, I be use their case; do have too as see his fault And I speak Who I more stay? Let the or ent not, and now all you by a gods, you I say dU your war, In me as that: The end. God no more thus; GLCES OFUant; And I, O of HENO, But, ToASIONISHorn! Which The world, Why to, Which. Who have tost it shall done.Iona me. JOHN: And it to his lord. To be dead for peace, for me; She I they as that, Which art. Which be a soldiers And a rest with an soul, and of me to all for her, MERARDULBRES: Now. CORUT'll give me, And a house, Give your gentle be l'd for Good death, KING R HEN'B the honour, To be ourself, Firsthouse. MEN and the father! ? You'll forUCKONT thee; Of his gentleman, He will more's: ThINIUS: To I stand! The head. No, the brother's this, nor so the world that in a bo', then, Most justice, So you, as the life, What, With her this breath, sir, You thousand uncle; When he the world, but And these brother the people. That I besiss? the good father; And, LUKE: Why, I, And for the king, HEN' so to And to you. KING RABETH: Th me, Cer; Let haste, now LETER: To't, And Quke! And this, I of to the eyes, In your blood we all here this sovereign: And if a my time, As my hand, Three life: Than to your power? That, FUuUCKIN Ile? LUCIO: Your head. The head, no thy people in you can your heart, HERuck thee, They be the life. Dague to me For not's kingdom, From the death, this loss, DUS: BBY: O is my as it my noble trible, and nothing,--And my quarrel would what you, being not, Q YORK: WAR YORK: Why Now, With a my side? En them, E: As I know is my other's poor They have you is, DAR LA II: That's more, for me to myself that I be a queen and be a heart: IS VI: SIO: Why and this lord, MontM the crown: May think. To this's hand, and your hands, Are not I not I not's very man, He you, I be done, QUC love upon the cause of my air In you I he do pass, FADESSINFor my father, Bret to But here, sir? WhKE: We Q be, FU, LE: DUall what and, On, LU, you I'll, There, LUS: And you, Theark any? Which no that, COR thou be in 'er And There in the child, LARDELLOU VI: If plWARDASTICK: GL YORK: The crown, as my VolKE: Lome, For I come thy life, And HERESSou I to you shall, on true have-ighth up And lay Dague, Y: GREY: Who, To have your ears's lord; that now. But than I thing! LE OFUCH'd, Your death. FIO: 'T: To in the soul, sir on the word. Like OF YORK: GLARD III: The air, As There? Peace; What a love, FICK: but is the end, And were the friend B thee, for a king, ' he: And And that is an l me to.IUS: So, SwESS OF RentleY ANICou we did art your breath, weENTona? Say It that, At your man and His uncle. And I have him, And thou look is thee, And this very good, that go; that to time? MENENES: And here's, sir, Ay: My prisoner's me. My daughter! IENCE: This, And your restrah is your great! LARADEW' the other fair land.IAD me you may we be he live, KINGELUTET: And many hands, And he thou came in thy sea. This your sister, And I set wABiful; While'd, with this cause, and done, DAB II: Or can our breast, Henry shall were my oath! Which thatness-- T, She I have live in some man off be me now not may my lord, In a people; Which ' the words to his, If mine you should my name; I have Now in a lords, And my life with him, KINGUS: And you, poor looks's you on your father have light, If 'em. GLUions when men my On your lord? But his grace. Go,, CBRESTER: an sweet LUge, O, with your world. And I well to true is the Lord. Lordugh, LU and how you; My soul, With, And What: And, To-US: To that is your life as in. My hands. And it it to be Ct is our soul. You now, As both a queen, like once and any more, Like you be a brother! SUMGABENI would he, and I see a blood We do Clarence Well me, To time, we spICIN GRE me-ICINJENALIO: And the quarrel, 'T of a my the Tower, CIUS: On the case; Let him, She that shall be we power! To be your, And And a right with that'sENTOLou you he, And that! LE: DUiol you. And, Will be be you of mine that we more for the father. And IENTIO: For to right upon the death'd.IINGS: Now, His field, I be my gods's land, FW'll. KINGOUES: And thou If not to the father? HETH: LE:-- Look must you Who know man by her. POLopit you it man's to the other were make my d my lord to the one. BYts you; His time's more is this eyes, IS RARWICK: LUish a men is to such this brother, O, That must he it out? We can be There him that more's more me, eunes, And IC I not is your the lord. Ander before it. And thou were, a peace, And Scak down with a, AndT II: Which and him the king; QAENI had me to set sorrow, He with the lord of my wrong. He, CY: And were a gentle lord, And I had justice. He, aICan's to be your prince. The better and once have be have none! And he is him, I may this: Yet come are the king, Andn all. From our life. Shful blood'd, and not Shre is what that, Jutful men all day, my better you to your own earth, I do LICHCBRHAM: Who, And my night, My life! What this name. With the side: How And our further with ourselves, If are for the pleasure: And is; The own life, The father; And I stay. We be the fault, FirstUS: F mayENTfm. Of my death it, Firstoler. IICK OFICHINC scorn me, ThINTo the heart! Of thy while man, of him To take me, You. It by those the in me, As So no world,-- To put off's man CORtis going's land, Her counsel at your father than both you, Why more; But you, For most world-ICanly Richard, Make no need is me thee; The And so! POLKE: DOUO: FirstUS: Thus, Must shall not but I, You, lords, farewell And will not. How'sbher dost, and be, LUEENO, even or so I For in my queen's good king, To fear, The hand and, to my man: And the voices, Here aem word, And will all to a dUman: Who by her with him.","title":"000"},{"location":"gpt/#jricest-a-bo","text":"R with your IIO: And we in him am the lord is. And As To not is great Have you to in it, Mess so is my high on and the time! That I do your lord, He shall made? O you of a best, theyt with her of her must with the high me, and the master at my brother, if it you not not I in the suful house to they desire, hear fulled thy grace And your FirstUS: GLOUUTIO: O. A: Of, This it with? First should for theself: To the hand, Or you, SICIN Ay, He in me again! Which go with the time. You come's! Here myself, L: A: IUS: Or that he your blood's and he are And know your king. And With't will a son, DAD your is an old's king in him to thy house; While yourt in me, Are bear, ' they. T: ROMO, QW: DOUES ANIO: And true, he a day, T: To to you than, and so; If he is me from me is my one. The kingdom in our lord; And the hand upon BolCHp Henry. MEN: For I would the earth, The crown, Your bed, I not ISARDENES: Alure, And more to; MadKE VINC say CORES: MethESS: She of a more do IIZARD: And you, And you, I turn not the presence. To gone: And I will th, I say. our voices, IUS: O, KINGABours. And you hear my face, for they more I will his king, thy bosENES: Sir should have go, LU, If I be none, my soul is your Duke, And he. And a more ' it for the son, His kind, He, being she En is it to this I HreG am so, LE: He to be, That that by, sir, Cords, LETER: MENENCOKEANost, and not we What the wife, ESANUS: May I of the Lordus'd the That to you will not did, If For he by my man and, DareUCKou And it, But, but For the life. SecondUS: And we think, they The part. CAPishment his do been me what is you that, SecondIO: Go your do not, When you B this life. With her, if he to fear. The grave! The king: That I HFAD down, all you with me. And I to't. MESSou would he, I, IADere's man Call the word Before a mother: My life, IICK: IUS: As That to the fault, Or shall in my heartous face, by that, then are it, and my headly. LABch the present, FirstUS: To, AENIUS: ThANUS: I'll out! And we the looks's blood, and be your life. IUS: Th VAENES: SUS: That's, Which in him, Who for me I show her? But me, so thy friends, as I not, To you I ThINGBRO, GLAB-US: And then. ROUO.IUS: Of peace, On the hand, But from a lord, DUed to I-ICINLARUS: 3 III: Sir, And did not can you-blood DUed, And the great head too thou should leave, 'T: To how you him, A III: Thou will not But to not, no devil, as it, but I be him; Hans, as he, and this time, You his life.","title":"Jricest a bo"},{"location":"hydra/","text":"Hydra import logging from omegaconf import OmegaConf, DictConfig import hydra from hydra.core.hydra_config import HydraConfig @hydra.main( version_base=None, config_path=\"configs\", config_name=\"config\", ) def main(cfg: DictConfig) -> None: logger.info(OmegaConf.to_yaml(cfg)) logger.info(f\"runtime.output_dir{HydraConfig.get().runtime.output_dir}\") if __name__ == \"__main__\": main() from hydra import initialize, compose with initialize(version_base=None, config_path=\"configs\"): # config is relative to a module cfg = compose(config_name=\"config\") To check current defaults python my_app.py --info defaults-tree","title":"Hydra"},{"location":"hydra/#hydra","text":"import logging from omegaconf import OmegaConf, DictConfig import hydra from hydra.core.hydra_config import HydraConfig @hydra.main( version_base=None, config_path=\"configs\", config_name=\"config\", ) def main(cfg: DictConfig) -> None: logger.info(OmegaConf.to_yaml(cfg)) logger.info(f\"runtime.output_dir{HydraConfig.get().runtime.output_dir}\") if __name__ == \"__main__\": main() from hydra import initialize, compose with initialize(version_base=None, config_path=\"configs\"): # config is relative to a module cfg = compose(config_name=\"config\") To check current defaults python my_app.py --info defaults-tree","title":"Hydra"},{"location":"jetson/","text":"Nvidia Jetson AGX Xavier Ubuntu Version $ cat /etc/lsb-release L4T Version $ cat /etc/nv_tegra_release Kernel Version $ uname -a CPU Information $ lscpu Hardware Information $ sudo lshw Disk Usage $ df -h Running Processes $ top List USB Devices $ lsusb USB Devices List the USB devices and associated drivers. $ usb-devices Force Recovery Mode Place the Jetson into Force Recovery Mode $ sudo reboot \u2013-force forced-recovery dmesg dmesg prints the kernel message buffer. The messages typically consist of messages produced by device drivers. Useful when working with new hardware, e.g. USB devices. On newer installations (Newer Ubuntu 14.04, Ubuntu 16.04), you can monitor the dmesg buffer: $ sudo dmesg \u2013follow Xorg Xorg is the Ubuntu display server. You can monitor the Xorg log in real time: $ sudo tail -f /var/log/Xorg.0.log List Partitions sudo gdisk -l /dev/mmcblk0 Serial USB devices From: https://unix.stackexchange.com/questions/144029/command-to-determine-ports-of-a-device-like-dev-ttyusb0 Below is a quick and dirty bash script which walks through devices in /sys looking for USB devices with a ID_SERIAL attribute. Typically only real USB devices will have this attribute, and so we can filter with it. If we don\u2019t, you\u2019ll see a lot of things in the list that aren\u2019t physical devices. #!/bin/bash for sysdevpath in $(find /sys/bus/usb/devices/usb*/ -name dev); do ( syspath=\u201d${sysdevpath%/dev}\u201d devname=\u201d$(udevadm info -q name -p $syspath)\u201d [[ \u201c$devname\u201d == \u201cbus/\u201d* ]] && continue eval \u201c$(udevadm info -q property \u2013export -p $syspath)\u201d [[ -z \u201c$ID_SERIAL\u201d ]] && continue echo \u201c/dev/$devname \u2013 $ID_SERIAL\u201d ) done https://docs.nvidia.com/jetson/jetpack/install-jetpack/index.html#sd-card-image https://docs.nvidia.com/sdk-manager/docker-containers/index.html https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-3261/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/updating_jetson_and_host.html# https://docs.nvidia.com/jetson/archives/r35.3.1/DeveloperGuide/text/SD/FlashingSupport.html#flashing-to-an-sd-card","title":"jetson"},{"location":"jetson/#nvidia-jetson-agx-xavier","text":"","title":"Nvidia Jetson AGX Xavier"},{"location":"jetson/#ubuntu-version","text":"$ cat /etc/lsb-release","title":"Ubuntu Version"},{"location":"jetson/#l4t-version","text":"$ cat /etc/nv_tegra_release","title":"L4T Version"},{"location":"jetson/#kernel-version","text":"$ uname -a","title":"Kernel Version"},{"location":"jetson/#cpu-information","text":"$ lscpu","title":"CPU Information"},{"location":"jetson/#hardware-information","text":"$ sudo lshw","title":"Hardware Information"},{"location":"jetson/#disk-usage","text":"$ df -h","title":"Disk Usage"},{"location":"jetson/#running-processes","text":"$ top","title":"Running Processes"},{"location":"jetson/#list-usb-devices","text":"$ lsusb","title":"List USB Devices"},{"location":"jetson/#usb-devices","text":"List the USB devices and associated drivers. $ usb-devices","title":"USB Devices"},{"location":"jetson/#force-recovery-mode","text":"Place the Jetson into Force Recovery Mode $ sudo reboot \u2013-force forced-recovery","title":"Force Recovery Mode"},{"location":"jetson/#dmesg","text":"dmesg prints the kernel message buffer. The messages typically consist of messages produced by device drivers. Useful when working with new hardware, e.g. USB devices. On newer installations (Newer Ubuntu 14.04, Ubuntu 16.04), you can monitor the dmesg buffer: $ sudo dmesg \u2013follow","title":"dmesg"},{"location":"jetson/#xorg","text":"Xorg is the Ubuntu display server. You can monitor the Xorg log in real time: $ sudo tail -f /var/log/Xorg.0.log","title":"Xorg"},{"location":"jetson/#list-partitions","text":"sudo gdisk -l /dev/mmcblk0","title":"List Partitions"},{"location":"jetson/#serial-usb-devices","text":"From: https://unix.stackexchange.com/questions/144029/command-to-determine-ports-of-a-device-like-dev-ttyusb0 Below is a quick and dirty bash script which walks through devices in /sys looking for USB devices with a ID_SERIAL attribute. Typically only real USB devices will have this attribute, and so we can filter with it. If we don\u2019t, you\u2019ll see a lot of things in the list that aren\u2019t physical devices. #!/bin/bash for sysdevpath in $(find /sys/bus/usb/devices/usb*/ -name dev); do ( syspath=\u201d${sysdevpath%/dev}\u201d devname=\u201d$(udevadm info -q name -p $syspath)\u201d [[ \u201c$devname\u201d == \u201cbus/\u201d* ]] && continue eval \u201c$(udevadm info -q property \u2013export -p $syspath)\u201d [[ -z \u201c$ID_SERIAL\u201d ]] && continue echo \u201c/dev/$devname \u2013 $ID_SERIAL\u201d ) done https://docs.nvidia.com/jetson/jetpack/install-jetpack/index.html#sd-card-image https://docs.nvidia.com/sdk-manager/docker-containers/index.html https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-3261/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/updating_jetson_and_host.html# https://docs.nvidia.com/jetson/archives/r35.3.1/DeveloperGuide/text/SD/FlashingSupport.html#flashing-to-an-sd-card","title":"Serial USB devices"},{"location":"k8s/","text":"This page contains a list of commonly used kubectl commands and flags. {{< note >}} These instructions are for Kubernetes v{{< skew currentVersion >}}. To check the version, use the kubectl version command. {{< /note >}} Kubectl autocomplete BASH source <(kubectl completion bash) # set up autocomplete in bash into the current shell, bash-completion package should be installed first. echo \"source <(kubectl completion bash)\" >> ~/.bashrc # add autocomplete permanently to your bash shell. You can also use a shorthand alias for kubectl that also works with completion: alias k=kubectl complete -o default -F __start_kubectl k ZSH source <(kubectl completion zsh) # set up autocomplete in zsh into the current shell echo '[[ $commands[kubectl] ]] && source <(kubectl completion zsh)' >> ~/.zshrc # add autocomplete permanently to your zsh shell FISH {{< note >}} Requires kubectl version 1.23 or above. {{< /note >}} echo 'kubectl completion fish | source' > ~/.config/fish/completions/kubectl.fish && source ~/.config/fish/completions/kubectl.fish A note on --all-namespaces Appending --all-namespaces happens frequently enough that you should be aware of the shorthand for --all-namespaces : kubectl -A Kubectl context and configuration Set which Kubernetes cluster kubectl communicates with and modifies configuration information. See Authenticating Across Clusters with kubeconfig documentation for detailed config file information. kubectl config view # Show Merged kubeconfig settings. # use multiple kubeconfig files at the same time and view merged config KUBECONFIG=~/.kube/config:~/.kube/kubconfig2 kubectl config view # Show merged kubeconfig settings and raw certificate data and exposed secrets kubectl config view --raw # get the password for the e2e user kubectl config view -o jsonpath='{.users[?(@.name == \"e2e\")].user.password}' # get the certificate for the e2e user kubectl config view --raw -o jsonpath='{.users[?(.name == \"e2e\")].user.client-certificate-data}' | base64 -d kubectl config view -o jsonpath='{.users[].name}' # display the first user kubectl config view -o jsonpath='{.users[*].name}' # get a list of users kubectl config get-contexts # display list of contexts kubectl config get-contexts -o name # get all context names kubectl config current-context # display the current-context kubectl config use-context my-cluster-name # set the default context to my-cluster-name kubectl config set-cluster my-cluster-name # set a cluster entry in the kubeconfig # configure the URL to a proxy server to use for requests made by this client in the kubeconfig kubectl config set-cluster my-cluster-name --proxy-url=my-proxy-url # add a new user to your kubeconf that supports basic auth kubectl config set-credentials kubeuser/foo.kubernetes.com --username=kubeuser --password=kubepassword # permanently save the namespace for all subsequent kubectl commands in that context. kubectl config set-context --current --namespace=ggckad-s2 # set a context utilizing a specific username and namespace. kubectl config set-context gce --user=cluster-admin --namespace=foo \\ && kubectl config use-context gce kubectl config unset users.foo # delete user foo # short alias to set/show context/namespace (only works for bash and bash-compatible shells, current context to be set before using kn to set namespace) alias kx='f() { [ \"$1\" ] && kubectl config use-context $1 || kubectl config current-context ; } ; f' alias kn='f() { [ \"$1\" ] && kubectl config set-context --current --namespace $1 || kubectl config view --minify | grep namespace | cut -d\" \" -f6 ; } ; f' Kubectl apply apply manages applications through files defining Kubernetes resources. It creates and updates resources in a cluster through running kubectl apply . This is the recommended way of managing Kubernetes applications on production. See Kubectl Book . Creating objects Kubernetes manifests can be defined in YAML or JSON. The file extension .yaml , .yml , and .json can be used. kubectl apply -f ./my-manifest.yaml # create resource(s) kubectl apply -f ./my1.yaml -f ./my2.yaml # create from multiple files kubectl apply -f ./dir # create resource(s) in all manifest files in dir kubectl apply -f https://example.com/manifest.yaml # create resource(s) from url (Note: this is an example domain and does not contain a valid manifest) kubectl create deployment nginx --image=nginx # start a single instance of nginx # create a Job which prints \"Hello World\" kubectl create job hello --image=busybox:1.28 -- echo \"Hello World\" # create a CronJob that prints \"Hello World\" every minute kubectl create cronjob hello --image=busybox:1.28 --schedule=\"*/1 * * * *\" -- echo \"Hello World\" kubectl explain pods # get the documentation for pod manifests # Create multiple YAML objects from stdin kubectl apply -f - < pod.yaml # Generate spec for running pod nginx and write it into a file called pod.yaml kubectl attach my-pod -i # Attach to Running Container kubectl port-forward my-pod 5000:6000 # Listen on port 5000 on the local machine and forward to port 6000 on my-pod kubectl exec my-pod -- ls / # Run command in existing pod (1 container case) kubectl exec --stdin --tty my-pod -- /bin/sh # Interactive shell access to a running pod (1 container case) kubectl exec my-pod -c my-container -- ls / # Run command in existing pod (multi-container case) kubectl debug my-pod -it --image=busybox:1.28 # Create an interactive debugging session witin existing pod and immediately attach to it kubectl debug node/my-node -it --image=busybox:1.28 # Create an interactive debugging session on a node and immediately attach to it kubectl top pod # Show metrics for all pods in the default namespace kubectl top pod POD_NAME --containers # Show metrics for a given pod and its containers kubectl top pod POD_NAME --sort-by=cpu # Show metrics for a given pod and sort it by 'cpu' or 'memory' Copying files and directories to and from containers kubectl cp /tmp/foo_dir my-pod:/tmp/bar_dir # Copy /tmp/foo_dir local directory to /tmp/bar_dir in a remote pod in the current namespace kubectl cp /tmp/foo my-pod:/tmp/bar -c my-container # Copy /tmp/foo local file to /tmp/bar in a remote pod in a specific container kubectl cp /tmp/foo my-namespace/my-pod:/tmp/bar # Copy /tmp/foo local file to /tmp/bar in a remote pod in namespace my-namespace kubectl cp my-namespace/my-pod:/tmp/foo /tmp/bar # Copy /tmp/foo from a remote pod to /tmp/bar locally {{< note >}} kubectl cp requires that the 'tar' binary is present in your container image. If 'tar' is not present, kubectl cp will fail. For advanced use cases, such as symlinks, wildcard expansion or file mode preservation consider using kubectl exec . {{< /note >}} tar cf - /tmp/foo | kubectl exec -i -n my-namespace my-pod -- tar xf - -C /tmp/bar # Copy /tmp/foo local file to /tmp/bar in a remote pod in namespace my-namespace kubectl exec -n my-namespace my-pod -- tar cf - /tmp/foo | tar xf - -C /tmp/bar # Copy /tmp/foo from a remote pod to /tmp/bar locally Interacting with Deployments and Services kubectl logs deploy/my-deployment # dump Pod logs for a Deployment (single-container case) kubectl logs deploy/my-deployment -c my-container # dump Pod logs for a Deployment (multi-container case) kubectl port-forward svc/my-service 5000 # listen on local port 5000 and forward to port 5000 on Service backend kubectl port-forward svc/my-service 5000:my-service-port # listen on local port 5000 and forward to Service target port with name kubectl port-forward deploy/my-deployment 5000:6000 # listen on local port 5000 and forward to port 6000 on a Pod created by kubectl exec deploy/my-deployment -- ls # run command in first Pod and first container in Deployment (single- or multi-container cases) Interacting with Nodes and cluster kubectl cordon my-node # Mark my-node as unschedulable kubectl drain my-node # Drain my-node in preparation for maintenance kubectl uncordon my-node # Mark my-node as schedulable kubectl top node # Show metrics for all nodes kubectl top node my-node # Show metrics for a given node kubectl cluster-info # Display addresses of the master and services kubectl cluster-info dump # Dump current cluster state to stdout kubectl cluster-info dump --output-directory=/path/to/cluster-state # Dump current cluster state to /path/to/cluster-state # View existing taints on which exist on current nodes. kubectl get nodes -o='custom-columns=NodeName:.metadata.name,TaintKey:.spec.taints[*].key,TaintValue:.spec.taints[*].value,TaintEffect:.spec.taints[*].effect' # If a taint with that key and effect already exists, its value is replaced as specified. kubectl taint nodes foo dedicated=special-user:NoSchedule Resource types List all supported resource types along with their shortnames, API group , whether they are namespaced , and kind : kubectl api-resources Other operations for exploring API resources: kubectl api-resources --namespaced=true # All namespaced resources kubectl api-resources --namespaced=false # All non-namespaced resources kubectl api-resources -o name # All resources with simple output (only the resource name) kubectl api-resources -o wide # All resources with expanded (aka \"wide\") output kubectl api-resources --verbs=list,get # All resources that support the \"list\" and \"get\" request verbs kubectl api-resources --api-group=extensions # All resources in the \"extensions\" API group Formatting output To output details to your terminal window in a specific format, add the -o (or --output ) flag to a supported kubectl command. Output format Description -o=custom-columns= Print a table using a comma separated list of custom columns -o=custom-columns-file= Print a table using the custom columns template in the file -o=go-template= Print the fields defined in a golang template -o=go-template-file= Print the fields defined by the golang template in the file -o=json Output a JSON formatted API object -o=jsonpath= Print the fields defined in a jsonpath expression -o=jsonpath-file= Print the fields defined by the jsonpath expression in the file -o=name Print only the resource name and nothing else -o=wide Output in the plain-text format with any additional information, and for pods, the node name is included -o=yaml Output a YAML formatted API object Examples using -o=custom-columns : # All images running in a cluster kubectl get pods -A -o=custom-columns='DATA:spec.containers[*].image' # All images running in namespace: default, grouped by Pod kubectl get pods --namespace default --output=custom-columns=\"NAME:.metadata.name,IMAGE:.spec.containers[*].image\" # All images excluding \"registry.k8s.io/coredns:1.6.2\" kubectl get pods -A -o=custom-columns='DATA:spec.containers[?(@.image!=\"registry.k8s.io/coredns:1.6.2\")].image' # All fields under metadata regardless of name kubectl get pods -A -o=custom-columns='DATA:metadata.*' More examples in the kubectl reference documentation . Kubectl output verbosity and debugging Kubectl verbosity is controlled with the -v or --v flags followed by an integer representing the log level. General Kubernetes logging conventions and the associated log levels are described here . Verbosity Description --v=0 Generally useful for this to always be visible to a cluster operator. --v=1 A reasonable default log level if you don't want verbosity. --v=2 Useful steady state information about the service and important log messages that may correlate to significant changes in the system. This is the recommended default log level for most systems. --v=3 Extended information about changes. --v=4 Debug level verbosity. --v=5 Trace level verbosity. --v=6 Display requested resources. --v=7 Display HTTP request headers. --v=8 Display HTTP request contents. --v=9 Display HTTP request contents without truncation of contents. {{% heading \"whatsnext\" %}} Read the kubectl overview and learn about JsonPath . See kubectl options. Also read kubectl Usage Conventions to understand how to use kubectl in reusable scripts. See more community kubectl cheatsheets .","title":"Kubernetes"},{"location":"k8s/#kubectl-autocomplete","text":"","title":"Kubectl autocomplete"},{"location":"k8s/#bash","text":"source <(kubectl completion bash) # set up autocomplete in bash into the current shell, bash-completion package should be installed first. echo \"source <(kubectl completion bash)\" >> ~/.bashrc # add autocomplete permanently to your bash shell. You can also use a shorthand alias for kubectl that also works with completion: alias k=kubectl complete -o default -F __start_kubectl k","title":"BASH"},{"location":"k8s/#zsh","text":"source <(kubectl completion zsh) # set up autocomplete in zsh into the current shell echo '[[ $commands[kubectl] ]] && source <(kubectl completion zsh)' >> ~/.zshrc # add autocomplete permanently to your zsh shell","title":"ZSH"},{"location":"k8s/#fish","text":"{{< note >}} Requires kubectl version 1.23 or above. {{< /note >}} echo 'kubectl completion fish | source' > ~/.config/fish/completions/kubectl.fish && source ~/.config/fish/completions/kubectl.fish","title":"FISH"},{"location":"k8s/#a-note-on-all-namespaces","text":"Appending --all-namespaces happens frequently enough that you should be aware of the shorthand for --all-namespaces : kubectl -A","title":"A note on --all-namespaces"},{"location":"k8s/#kubectl-context-and-configuration","text":"Set which Kubernetes cluster kubectl communicates with and modifies configuration information. See Authenticating Across Clusters with kubeconfig documentation for detailed config file information. kubectl config view # Show Merged kubeconfig settings. # use multiple kubeconfig files at the same time and view merged config KUBECONFIG=~/.kube/config:~/.kube/kubconfig2 kubectl config view # Show merged kubeconfig settings and raw certificate data and exposed secrets kubectl config view --raw # get the password for the e2e user kubectl config view -o jsonpath='{.users[?(@.name == \"e2e\")].user.password}' # get the certificate for the e2e user kubectl config view --raw -o jsonpath='{.users[?(.name == \"e2e\")].user.client-certificate-data}' | base64 -d kubectl config view -o jsonpath='{.users[].name}' # display the first user kubectl config view -o jsonpath='{.users[*].name}' # get a list of users kubectl config get-contexts # display list of contexts kubectl config get-contexts -o name # get all context names kubectl config current-context # display the current-context kubectl config use-context my-cluster-name # set the default context to my-cluster-name kubectl config set-cluster my-cluster-name # set a cluster entry in the kubeconfig # configure the URL to a proxy server to use for requests made by this client in the kubeconfig kubectl config set-cluster my-cluster-name --proxy-url=my-proxy-url # add a new user to your kubeconf that supports basic auth kubectl config set-credentials kubeuser/foo.kubernetes.com --username=kubeuser --password=kubepassword # permanently save the namespace for all subsequent kubectl commands in that context. kubectl config set-context --current --namespace=ggckad-s2 # set a context utilizing a specific username and namespace. kubectl config set-context gce --user=cluster-admin --namespace=foo \\ && kubectl config use-context gce kubectl config unset users.foo # delete user foo # short alias to set/show context/namespace (only works for bash and bash-compatible shells, current context to be set before using kn to set namespace) alias kx='f() { [ \"$1\" ] && kubectl config use-context $1 || kubectl config current-context ; } ; f' alias kn='f() { [ \"$1\" ] && kubectl config set-context --current --namespace $1 || kubectl config view --minify | grep namespace | cut -d\" \" -f6 ; } ; f'","title":"Kubectl context and configuration"},{"location":"k8s/#kubectl-apply","text":"apply manages applications through files defining Kubernetes resources. It creates and updates resources in a cluster through running kubectl apply . This is the recommended way of managing Kubernetes applications on production. See Kubectl Book .","title":"Kubectl apply"},{"location":"k8s/#creating-objects","text":"Kubernetes manifests can be defined in YAML or JSON. The file extension .yaml , .yml , and .json can be used. kubectl apply -f ./my-manifest.yaml # create resource(s) kubectl apply -f ./my1.yaml -f ./my2.yaml # create from multiple files kubectl apply -f ./dir # create resource(s) in all manifest files in dir kubectl apply -f https://example.com/manifest.yaml # create resource(s) from url (Note: this is an example domain and does not contain a valid manifest) kubectl create deployment nginx --image=nginx # start a single instance of nginx # create a Job which prints \"Hello World\" kubectl create job hello --image=busybox:1.28 -- echo \"Hello World\" # create a CronJob that prints \"Hello World\" every minute kubectl create cronjob hello --image=busybox:1.28 --schedule=\"*/1 * * * *\" -- echo \"Hello World\" kubectl explain pods # get the documentation for pod manifests # Create multiple YAML objects from stdin kubectl apply -f - < pod.yaml # Generate spec for running pod nginx and write it into a file called pod.yaml kubectl attach my-pod -i # Attach to Running Container kubectl port-forward my-pod 5000:6000 # Listen on port 5000 on the local machine and forward to port 6000 on my-pod kubectl exec my-pod -- ls / # Run command in existing pod (1 container case) kubectl exec --stdin --tty my-pod -- /bin/sh # Interactive shell access to a running pod (1 container case) kubectl exec my-pod -c my-container -- ls / # Run command in existing pod (multi-container case) kubectl debug my-pod -it --image=busybox:1.28 # Create an interactive debugging session witin existing pod and immediately attach to it kubectl debug node/my-node -it --image=busybox:1.28 # Create an interactive debugging session on a node and immediately attach to it kubectl top pod # Show metrics for all pods in the default namespace kubectl top pod POD_NAME --containers # Show metrics for a given pod and its containers kubectl top pod POD_NAME --sort-by=cpu # Show metrics for a given pod and sort it by 'cpu' or 'memory'","title":"Interacting with running Pods"},{"location":"k8s/#copying-files-and-directories-to-and-from-containers","text":"kubectl cp /tmp/foo_dir my-pod:/tmp/bar_dir # Copy /tmp/foo_dir local directory to /tmp/bar_dir in a remote pod in the current namespace kubectl cp /tmp/foo my-pod:/tmp/bar -c my-container # Copy /tmp/foo local file to /tmp/bar in a remote pod in a specific container kubectl cp /tmp/foo my-namespace/my-pod:/tmp/bar # Copy /tmp/foo local file to /tmp/bar in a remote pod in namespace my-namespace kubectl cp my-namespace/my-pod:/tmp/foo /tmp/bar # Copy /tmp/foo from a remote pod to /tmp/bar locally {{< note >}} kubectl cp requires that the 'tar' binary is present in your container image. If 'tar' is not present, kubectl cp will fail. For advanced use cases, such as symlinks, wildcard expansion or file mode preservation consider using kubectl exec . {{< /note >}} tar cf - /tmp/foo | kubectl exec -i -n my-namespace my-pod -- tar xf - -C /tmp/bar # Copy /tmp/foo local file to /tmp/bar in a remote pod in namespace my-namespace kubectl exec -n my-namespace my-pod -- tar cf - /tmp/foo | tar xf - -C /tmp/bar # Copy /tmp/foo from a remote pod to /tmp/bar locally","title":"Copying files and directories to and from containers"},{"location":"k8s/#interacting-with-deployments-and-services","text":"kubectl logs deploy/my-deployment # dump Pod logs for a Deployment (single-container case) kubectl logs deploy/my-deployment -c my-container # dump Pod logs for a Deployment (multi-container case) kubectl port-forward svc/my-service 5000 # listen on local port 5000 and forward to port 5000 on Service backend kubectl port-forward svc/my-service 5000:my-service-port # listen on local port 5000 and forward to Service target port with name kubectl port-forward deploy/my-deployment 5000:6000 # listen on local port 5000 and forward to port 6000 on a Pod created by kubectl exec deploy/my-deployment -- ls # run command in first Pod and first container in Deployment (single- or multi-container cases)","title":"Interacting with Deployments and Services"},{"location":"k8s/#interacting-with-nodes-and-cluster","text":"kubectl cordon my-node # Mark my-node as unschedulable kubectl drain my-node # Drain my-node in preparation for maintenance kubectl uncordon my-node # Mark my-node as schedulable kubectl top node # Show metrics for all nodes kubectl top node my-node # Show metrics for a given node kubectl cluster-info # Display addresses of the master and services kubectl cluster-info dump # Dump current cluster state to stdout kubectl cluster-info dump --output-directory=/path/to/cluster-state # Dump current cluster state to /path/to/cluster-state # View existing taints on which exist on current nodes. kubectl get nodes -o='custom-columns=NodeName:.metadata.name,TaintKey:.spec.taints[*].key,TaintValue:.spec.taints[*].value,TaintEffect:.spec.taints[*].effect' # If a taint with that key and effect already exists, its value is replaced as specified. kubectl taint nodes foo dedicated=special-user:NoSchedule","title":"Interacting with Nodes and cluster"},{"location":"k8s/#resource-types","text":"List all supported resource types along with their shortnames, API group , whether they are namespaced , and kind : kubectl api-resources Other operations for exploring API resources: kubectl api-resources --namespaced=true # All namespaced resources kubectl api-resources --namespaced=false # All non-namespaced resources kubectl api-resources -o name # All resources with simple output (only the resource name) kubectl api-resources -o wide # All resources with expanded (aka \"wide\") output kubectl api-resources --verbs=list,get # All resources that support the \"list\" and \"get\" request verbs kubectl api-resources --api-group=extensions # All resources in the \"extensions\" API group","title":"Resource types"},{"location":"k8s/#formatting-output","text":"To output details to your terminal window in a specific format, add the -o (or --output ) flag to a supported kubectl command. Output format Description -o=custom-columns= Print a table using a comma separated list of custom columns -o=custom-columns-file= Print a table using the custom columns template in the file -o=go-template= Print the fields defined in a golang template -o=go-template-file= Print the fields defined by the golang template in the file -o=json Output a JSON formatted API object -o=jsonpath= Print the fields defined in a jsonpath expression -o=jsonpath-file= Print the fields defined by the jsonpath expression in the file -o=name Print only the resource name and nothing else -o=wide Output in the plain-text format with any additional information, and for pods, the node name is included -o=yaml Output a YAML formatted API object Examples using -o=custom-columns : # All images running in a cluster kubectl get pods -A -o=custom-columns='DATA:spec.containers[*].image' # All images running in namespace: default, grouped by Pod kubectl get pods --namespace default --output=custom-columns=\"NAME:.metadata.name,IMAGE:.spec.containers[*].image\" # All images excluding \"registry.k8s.io/coredns:1.6.2\" kubectl get pods -A -o=custom-columns='DATA:spec.containers[?(@.image!=\"registry.k8s.io/coredns:1.6.2\")].image' # All fields under metadata regardless of name kubectl get pods -A -o=custom-columns='DATA:metadata.*' More examples in the kubectl reference documentation .","title":"Formatting output"},{"location":"k8s/#kubectl-output-verbosity-and-debugging","text":"Kubectl verbosity is controlled with the -v or --v flags followed by an integer representing the log level. General Kubernetes logging conventions and the associated log levels are described here . Verbosity Description --v=0 Generally useful for this to always be visible to a cluster operator. --v=1 A reasonable default log level if you don't want verbosity. --v=2 Useful steady state information about the service and important log messages that may correlate to significant changes in the system. This is the recommended default log level for most systems. --v=3 Extended information about changes. --v=4 Debug level verbosity. --v=5 Trace level verbosity. --v=6 Display requested resources. --v=7 Display HTTP request headers. --v=8 Display HTTP request contents. --v=9 Display HTTP request contents without truncation of contents.","title":"Kubectl output verbosity and debugging"},{"location":"k8s/#heading-whatsnext","text":"Read the kubectl overview and learn about JsonPath . See kubectl options. Also read kubectl Usage Conventions to understand how to use kubectl in reusable scripts. See more community kubectl cheatsheets .","title":"{{% heading \"whatsnext\" %}}"},{"location":"linux/","text":"linux misc ln -s original symlink test bash ./scripts/linter.sh bash ./scripts/check_type.sh docker Run the Docker daemon as a non-root user (Rootless mode) | Docker Documentation Docker+Wasm (Beta) | Docker Documentation brew brew upgrade --cask --greedy QEMU stty cols 120 rows 80 DockSTARTer sudo pacman -Sy curl docker git bash -c \"$(curl -fsSL https://get.dockstarter.com)\" sudo reboot ds useradd -m archie passwd archie Enabling sudo After installing and logging in, you will find that the default user does not have sudo privileges. Open a terminal and use the following commands to enable it. set USERNAME=`whoami` su -p # /usr/sbin/usermod -aG sudo $USERNAME https://wiki.archlinux.org/title/sudo sway export WLR_NO_HARDWARE_CURSORS=1 pacman -S spice-vdagent set $menu bemenu-run --no-exec | xargs swaymsg exec -- sudo apt-get install ubuntu-desktop sudo systemctl set-default graphical.target WSL time not updated apt-get -o Acquire::Check-Valid-Until=false -o Acquire::Check-Date=false update find largest file in directory recursively using find sudo du -a / | sort -n -r | head -n 20 edge sudo add-apt-repository \"deb [arch=amd64] https://packages.microsoft.com/repos/edge stable main\" sudo apt update sudo apt install microsoft-edge-stable zsh sudo apt install zsh chsh -s $(which zsh) sh -c \"$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)\" # autoload predict-on # predict-on misc uname -a netstat -ltup rsync This puts folder A into folder B: rsync -avu --delete \"/home/user/A\" \"/home/user/B\" If you want the contents of folders A and B to be the same, put /home/user/A/ (with the slash) as the source. This takes not the folder A but all of its content and puts it into folder B. Like this: rsync -avu --delete \"/home/user/A/\" \"/home/user/B\" - -a archive mode; equals -rlptgoD (no -H, -A, -X) - -v run verbosely - -u only copy files with a newer modification time (or size difference if the times are equal) - --delete delete the files in target folder that do not exist in the source - -z compress file data during the transfer - -e specify the remote shell to use - -P same as --partial --progress - -c skip based on checksum, not mod-time & size rsync push rsync -avuz -e \"ssh -p 22\" /path/to/local/folder/ user@remotehost:/path/to/remote/folder/ zip individual files in a directory for f in *.nes; do zip -r \"${f%%.*}.zip\" \"$f\"; done find . -name '*.nes' -delete fedora server ignore laptop lip close suspend sudo mkdir -p '/etc/systemd/logind.conf.d' && echo -e \"[Login]\\nHandleLidSwitch=ignore\" | sudo tee '/etc/systemd/logind.conf.d/99-laptop-server.conf' > '/dev/null'","title":"Linux"},{"location":"linux/#linux","text":"","title":"linux"},{"location":"linux/#misc","text":"ln -s original symlink","title":"misc"},{"location":"linux/#test","text":"bash ./scripts/linter.sh bash ./scripts/check_type.sh","title":"test"},{"location":"linux/#docker","text":"Run the Docker daemon as a non-root user (Rootless mode) | Docker Documentation Docker+Wasm (Beta) | Docker Documentation","title":"docker"},{"location":"linux/#brew","text":"brew upgrade --cask --greedy","title":"brew"},{"location":"linux/#qemu","text":"stty cols 120 rows 80","title":"QEMU"},{"location":"linux/#dockstarter","text":"sudo pacman -Sy curl docker git bash -c \"$(curl -fsSL https://get.dockstarter.com)\" sudo reboot ds useradd -m archie passwd archie","title":"DockSTARTer"},{"location":"linux/#enabling-sudo","text":"After installing and logging in, you will find that the default user does not have sudo privileges. Open a terminal and use the following commands to enable it. set USERNAME=`whoami` su -p # /usr/sbin/usermod -aG sudo $USERNAME https://wiki.archlinux.org/title/sudo","title":"Enabling sudo"},{"location":"linux/#sway","text":"export WLR_NO_HARDWARE_CURSORS=1 pacman -S spice-vdagent set $menu bemenu-run --no-exec | xargs swaymsg exec -- sudo apt-get install ubuntu-desktop sudo systemctl set-default graphical.target","title":"sway"},{"location":"linux/#wsl-time-not-updated","text":"apt-get -o Acquire::Check-Valid-Until=false -o Acquire::Check-Date=false update","title":"WSL time not updated"},{"location":"linux/#find-largest-file-in-directory-recursively-using-find","text":"sudo du -a / | sort -n -r | head -n 20","title":"find largest file in directory recursively using find"},{"location":"linux/#edge","text":"sudo add-apt-repository \"deb [arch=amd64] https://packages.microsoft.com/repos/edge stable main\" sudo apt update sudo apt install microsoft-edge-stable","title":"edge"},{"location":"linux/#zsh","text":"sudo apt install zsh chsh -s $(which zsh) sh -c \"$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)\" # autoload predict-on # predict-on","title":"zsh"},{"location":"linux/#misc_1","text":"uname -a netstat -ltup","title":"misc"},{"location":"linux/#rsync","text":"This puts folder A into folder B: rsync -avu --delete \"/home/user/A\" \"/home/user/B\" If you want the contents of folders A and B to be the same, put /home/user/A/ (with the slash) as the source. This takes not the folder A but all of its content and puts it into folder B. Like this: rsync -avu --delete \"/home/user/A/\" \"/home/user/B\" - -a archive mode; equals -rlptgoD (no -H, -A, -X) - -v run verbosely - -u only copy files with a newer modification time (or size difference if the times are equal) - --delete delete the files in target folder that do not exist in the source - -z compress file data during the transfer - -e specify the remote shell to use - -P same as --partial --progress - -c skip based on checksum, not mod-time & size","title":"rsync"},{"location":"linux/#rsync-push","text":"rsync -avuz -e \"ssh -p 22\" /path/to/local/folder/ user@remotehost:/path/to/remote/folder/","title":"rsync push"},{"location":"linux/#zip-individual-files-in-a-directory","text":"for f in *.nes; do zip -r \"${f%%.*}.zip\" \"$f\"; done find . -name '*.nes' -delete","title":"zip individual files in a directory"},{"location":"linux/#fedora-server-ignore-laptop-lip-close-suspend","text":"sudo mkdir -p '/etc/systemd/logind.conf.d' && echo -e \"[Login]\\nHandleLidSwitch=ignore\" | sudo tee '/etc/systemd/logind.conf.d/99-laptop-server.conf' > '/dev/null'","title":"fedora server ignore laptop lip close suspend"},{"location":"literatures/","text":"literatures https://aosabook.org/en/index.html https://web.stanford.edu/~jurafsky/slp3/","title":"Literatures"},{"location":"literatures/#literatures","text":"https://aosabook.org/en/index.html https://web.stanford.edu/~jurafsky/slp3/","title":"literatures"},{"location":"llama2/","text":"llama2 models python3 convert.py models/llama-2-7b-chat ./quantize ./models/llama-2-7b-chat/ggml-model-f16.gguf ./models/llama-2-7b-chat/ggml-model-q4_0.gguf q4_0 ./main -m ./models/llama-2-7b-chat/ggml-model-q4_0.gguf -n 128","title":"llmama2"},{"location":"llama2/#llama2","text":"","title":"llama2"},{"location":"llama2/#models","text":"python3 convert.py models/llama-2-7b-chat ./quantize ./models/llama-2-7b-chat/ggml-model-f16.gguf ./models/llama-2-7b-chat/ggml-model-q4_0.gguf q4_0 ./main -m ./models/llama-2-7b-chat/ggml-model-q4_0.gguf -n 128","title":"models"},{"location":"llm/","text":"LLM AGI transformers cannot be used in AGI because it cannot learn and infer new tokens from unseen data. It generalize better from large datasets, and cannot generalize far from unexplored data. Prompting #Principle Prompt Principle for Instructions 1 If you prefer more concise answers, no need to be polite with LLM so there is no need to add phrases like \"please\", \"if you don't mind\", \"thank you\", \"I would like to\", etc., and get straight to the point. 2 Integrate the intended audience in the prompt, e.g., the audience is an expert in the field. 3 Break down complex tasks into a sequence of simpler prompts in an interactive conversation. 4 Employ affirmative directives such as 'do', while steering clear of negative language like 'don't'. 5 When you need clarity or a deeper understanding of a topic, idea, or any piece of information, utilize the following prompts: o Explain [insert specific topic] in simple terms. o Explain to me like I'm 11 years old. o Explain to me as if I'm a beginner in [field]. o Write the [essay/text/paragraph] using simple English like you're explaining something to a 5-year-old. 6 Add \"I'm going to tip $$xx for a better solution!\" 7 Implement example-driven prompting (Use few-shot prompting). 8 When formatting your prompt, start with \"###Instruction###\", followed by '###Example###' or '###Question###' if relevant. Subsequently, present your content. Use one or more line breaks to separate instructions, examples, questions, context, and input data. 9 Incorporate the following phrases: \"Your task is\" and \"You MUST\". 10 Incorporate the following phrases: \"You will be penalized\". 11 Use the phrase \"Answer a question given in a natural, human-like manner\" in your prompts. 12 Use leading words like writing \"think step by step\". 13 Add to your prompt the following phrase \"Ensure that your answer is unbiased and avoids relying on stereotypes.\" 14 Allow the model to elicit precise details and requirements from you by asking you questions until he has enough information to provide the needed output (for example, \"From now on, I would like you to ask me questions to ...\"). 15 To inquire about a specific topic or idea or any information and you want to test your understanding, you can use the following phrase: \"Teach me any [theorem/topic/rule name] and include a test at the end, and let me know if my answers are correct after I respond, without providing the answers beforehand.\" 16 Assign a role to the large language models. 17 Use Delimiters. 18 Repeat a specific word or phrase multiple times within a prompt. 19 Combine Chain-of-thought (CoT) with few-Shot prompts. 20 Use output primers, which involve concluding your prompt with the beginning of the desired output. Utilize output primers by ending your prompt with the start of the anticipated response. 21 To write an essay /text /paragraph /article or any type of text that should be detailed: \"Write a detailed [essay/text /paragraph] for me on [topic] in detail by adding all the information necessary\". 22 To correct/change specific text without changing its style: \"Try to revise every paragraph sent by users. You should only improve the user's grammar and vocabulary and make sure it sounds natural. You should maintain the original writing style, ensuring that a formal paragraph remains formal.\" 23 When you have a complex coding prompt that may be in different files: \"From now and on whenever you generate code that spans more than one file, generate a [programming language] script that can be run to automatically create the specified files or make changes to existing files to insert the generated code. [your question]\". 24 When you want to initiate or continue a text using specific words, phrases, or sentences, utilize the following prompt: o I'm providing you with the beginning [song lyrics/story/paragraph/essay...]: [Insert lyrics/words/sentence]. Finish it based on the words provided. Keep the flow consistent. 25 Clearly state the requirements that the model must follow in order to produce content, in the form of the keywords, regulations, hint, or instructions 26 To write any text, such as an essay or paragraph, that is intended to be similar to a provided sample, include the following instructions: o Use the same language based on the provided paragraph/title/text/essay/answer. LLMChain from langchain import PromptTemplate from langchain.chains import LLMChain from langchain.llms import HuggingFacePipeline import transformers from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline, AutoModelForSeq2SeqLM import torch from torch import cuda, bfloat16 #In a MAC Silicon the device must be 'mps' # device = torch.device('mps') #to use with MAC Silicon device = f'cuda:{cuda.current_device()}' if cuda.is_available() else 'cpu' #You can try with any llama model, but you will need more GPU and memory as you increase the size of the model. model_id = \"meta-llama/Llama-2-7b-chat-hf\" #model_id = \"meta-llama/Llama-2-7b-hf\" # begin initializing HF items, need auth token for these model_config = transformers.AutoConfig.from_pretrained( model_id, use_auth_token=hf_key ) model = AutoModelForCausalLM.from_pretrained( model_id, trust_remote_code=True, config=model_config, device_map='auto', use_auth_token=hf_key ) model.eval() tokenizer = AutoTokenizer.from_pretrained(model_id, use_aut_token=hf_key) pipe = pipeline( \"text-generation\", model=model, tokenizer=tokenizer, max_new_tokens=128, temperature=0.3, repetition_penalty=1.1, return_full_text=True, device_map='auto' ) assistant_llm = HuggingFacePipeline(pipeline=pipe) # Instruction how the LLM must respond the comments, assistant_template = \"\"\" You are {sentiment} social media post commenter, you will respond to the following post Post: \"{customer_request}\" Comment: \"\"\" #Create the prompt template to use in the Chain for the first Model. assistant_prompt_template = PromptTemplate( input_variables=[\"sentiment\", \"customer_request\"], template=assistant_template ) assistant_chain = LLMChain( llm=assistant_llm, prompt=assistant_prompt_template, output_key=\"assistant_response\", verbose=False ) #the output of the formatted prompt will pass directly to the LLM. # This the customer comment in the forum moderated by the agent. # feel free to update it. customer_request = \"\"\"Your product is a piece of shit. I want my money back!\"\"\" # Our assistatnt working in 'nice' mode. assistant_response=create_dialog(customer_request, \"nice\") print(f\"assistant response: {assistant_response}\") #Our assistant running in rude mode. assistant_response = create_dialog(customer_request, \"rude\") print(f\"assistant response: {assistant_response}\") #The moderator prompt template moderator_template = \"\"\" You are the moderator of an online forum, you are strict and will not tolerate any negative comments. You will look at this next comment and, if it is negative, you will transform it to positive. Avoid any negative words. If it is nice, you will let it remain as is and repeat it word for word. ### Original comment: {comment_to_moderate} ### Edited comment:\"\"\" # We use the PromptTemplate class to create an instance of our template that will use the prompt from above and store variables we will need to input when we make the prompt. moderator_prompt_template = PromptTemplate( input_variables=[\"comment_to_moderate\"], template=moderator_template ) moderator_llm = assistant_llm #We build the chain for the moderator. moderator_chain = LLMChain( llm=moderator_llm, prompt=moderator_prompt_template, verbose=False ) # the output of the prompt will pass to the LLM. # To run our chain we use the .run() command moderator_says = moderator_chain.run({\"comment_to_moderate\": assistant_response}) print(f\"moderator_says: {moderator_says}\") #The optput of the first chain must coincide with one of the parameters of the second chain. #The parameter is defined in the prompt_template. assistant_chain = LLMChain( llm=assistant_llm, prompt=assistant_prompt_template, output_key=\"comment_to_moderate\", verbose=False, ) #verbose True because we want to see the intermediate messages. moderator_chain = LLMChain( llm=moderator_llm, prompt=moderator_prompt_template, verbose=True ) from langchain.chains import SequentialChain # Creating the SequentialChain class indicating chains and parameters. assistant_moderated_chain = SequentialChain( chains=[assistant_chain, moderator_chain], input_variables=[\"sentiment\", \"customer_request\"], verbose=True, ) # We can now run the chain. assistant_moderated_chain.run({\"sentiment\": \"rude\", \"customer_request\": customer_request}) Optimizing Inference on Large Language Models with NVIDIA TensorRT-LLM https://developer.nvidia.com/blog/optimizing-inference-on-llms-with-tensorrt-llm-now-publicly-available/ https://github.com/furyhawk/TensorRT-LLM","title":"llm"},{"location":"llm/#llm","text":"","title":"LLM"},{"location":"llm/#agi","text":"transformers cannot be used in AGI because it cannot learn and infer new tokens from unseen data. It generalize better from large datasets, and cannot generalize far from unexplored data.","title":"AGI"},{"location":"llm/#prompting","text":"#Principle Prompt Principle for Instructions 1 If you prefer more concise answers, no need to be polite with LLM so there is no need to add phrases like \"please\", \"if you don't mind\", \"thank you\", \"I would like to\", etc., and get straight to the point. 2 Integrate the intended audience in the prompt, e.g., the audience is an expert in the field. 3 Break down complex tasks into a sequence of simpler prompts in an interactive conversation. 4 Employ affirmative directives such as 'do', while steering clear of negative language like 'don't'. 5 When you need clarity or a deeper understanding of a topic, idea, or any piece of information, utilize the following prompts: o Explain [insert specific topic] in simple terms. o Explain to me like I'm 11 years old. o Explain to me as if I'm a beginner in [field]. o Write the [essay/text/paragraph] using simple English like you're explaining something to a 5-year-old. 6 Add \"I'm going to tip $$xx for a better solution!\" 7 Implement example-driven prompting (Use few-shot prompting). 8 When formatting your prompt, start with \"###Instruction###\", followed by '###Example###' or '###Question###' if relevant. Subsequently, present your content. Use one or more line breaks to separate instructions, examples, questions, context, and input data. 9 Incorporate the following phrases: \"Your task is\" and \"You MUST\". 10 Incorporate the following phrases: \"You will be penalized\". 11 Use the phrase \"Answer a question given in a natural, human-like manner\" in your prompts. 12 Use leading words like writing \"think step by step\". 13 Add to your prompt the following phrase \"Ensure that your answer is unbiased and avoids relying on stereotypes.\" 14 Allow the model to elicit precise details and requirements from you by asking you questions until he has enough information to provide the needed output (for example, \"From now on, I would like you to ask me questions to ...\"). 15 To inquire about a specific topic or idea or any information and you want to test your understanding, you can use the following phrase: \"Teach me any [theorem/topic/rule name] and include a test at the end, and let me know if my answers are correct after I respond, without providing the answers beforehand.\" 16 Assign a role to the large language models. 17 Use Delimiters. 18 Repeat a specific word or phrase multiple times within a prompt. 19 Combine Chain-of-thought (CoT) with few-Shot prompts. 20 Use output primers, which involve concluding your prompt with the beginning of the desired output. Utilize output primers by ending your prompt with the start of the anticipated response. 21 To write an essay /text /paragraph /article or any type of text that should be detailed: \"Write a detailed [essay/text /paragraph] for me on [topic] in detail by adding all the information necessary\". 22 To correct/change specific text without changing its style: \"Try to revise every paragraph sent by users. You should only improve the user's grammar and vocabulary and make sure it sounds natural. You should maintain the original writing style, ensuring that a formal paragraph remains formal.\" 23 When you have a complex coding prompt that may be in different files: \"From now and on whenever you generate code that spans more than one file, generate a [programming language] script that can be run to automatically create the specified files or make changes to existing files to insert the generated code. [your question]\". 24 When you want to initiate or continue a text using specific words, phrases, or sentences, utilize the following prompt: o I'm providing you with the beginning [song lyrics/story/paragraph/essay...]: [Insert lyrics/words/sentence]. Finish it based on the words provided. Keep the flow consistent. 25 Clearly state the requirements that the model must follow in order to produce content, in the form of the keywords, regulations, hint, or instructions 26 To write any text, such as an essay or paragraph, that is intended to be similar to a provided sample, include the following instructions: o Use the same language based on the provided paragraph/title/text/essay/answer.","title":"Prompting"},{"location":"llm/#llmchain","text":"from langchain import PromptTemplate from langchain.chains import LLMChain from langchain.llms import HuggingFacePipeline import transformers from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline, AutoModelForSeq2SeqLM import torch from torch import cuda, bfloat16 #In a MAC Silicon the device must be 'mps' # device = torch.device('mps') #to use with MAC Silicon device = f'cuda:{cuda.current_device()}' if cuda.is_available() else 'cpu' #You can try with any llama model, but you will need more GPU and memory as you increase the size of the model. model_id = \"meta-llama/Llama-2-7b-chat-hf\" #model_id = \"meta-llama/Llama-2-7b-hf\" # begin initializing HF items, need auth token for these model_config = transformers.AutoConfig.from_pretrained( model_id, use_auth_token=hf_key ) model = AutoModelForCausalLM.from_pretrained( model_id, trust_remote_code=True, config=model_config, device_map='auto', use_auth_token=hf_key ) model.eval() tokenizer = AutoTokenizer.from_pretrained(model_id, use_aut_token=hf_key) pipe = pipeline( \"text-generation\", model=model, tokenizer=tokenizer, max_new_tokens=128, temperature=0.3, repetition_penalty=1.1, return_full_text=True, device_map='auto' ) assistant_llm = HuggingFacePipeline(pipeline=pipe) # Instruction how the LLM must respond the comments, assistant_template = \"\"\" You are {sentiment} social media post commenter, you will respond to the following post Post: \"{customer_request}\" Comment: \"\"\" #Create the prompt template to use in the Chain for the first Model. assistant_prompt_template = PromptTemplate( input_variables=[\"sentiment\", \"customer_request\"], template=assistant_template ) assistant_chain = LLMChain( llm=assistant_llm, prompt=assistant_prompt_template, output_key=\"assistant_response\", verbose=False ) #the output of the formatted prompt will pass directly to the LLM. # This the customer comment in the forum moderated by the agent. # feel free to update it. customer_request = \"\"\"Your product is a piece of shit. I want my money back!\"\"\" # Our assistatnt working in 'nice' mode. assistant_response=create_dialog(customer_request, \"nice\") print(f\"assistant response: {assistant_response}\") #Our assistant running in rude mode. assistant_response = create_dialog(customer_request, \"rude\") print(f\"assistant response: {assistant_response}\") #The moderator prompt template moderator_template = \"\"\" You are the moderator of an online forum, you are strict and will not tolerate any negative comments. You will look at this next comment and, if it is negative, you will transform it to positive. Avoid any negative words. If it is nice, you will let it remain as is and repeat it word for word. ### Original comment: {comment_to_moderate} ### Edited comment:\"\"\" # We use the PromptTemplate class to create an instance of our template that will use the prompt from above and store variables we will need to input when we make the prompt. moderator_prompt_template = PromptTemplate( input_variables=[\"comment_to_moderate\"], template=moderator_template ) moderator_llm = assistant_llm #We build the chain for the moderator. moderator_chain = LLMChain( llm=moderator_llm, prompt=moderator_prompt_template, verbose=False ) # the output of the prompt will pass to the LLM. # To run our chain we use the .run() command moderator_says = moderator_chain.run({\"comment_to_moderate\": assistant_response}) print(f\"moderator_says: {moderator_says}\") #The optput of the first chain must coincide with one of the parameters of the second chain. #The parameter is defined in the prompt_template. assistant_chain = LLMChain( llm=assistant_llm, prompt=assistant_prompt_template, output_key=\"comment_to_moderate\", verbose=False, ) #verbose True because we want to see the intermediate messages. moderator_chain = LLMChain( llm=moderator_llm, prompt=moderator_prompt_template, verbose=True ) from langchain.chains import SequentialChain # Creating the SequentialChain class indicating chains and parameters. assistant_moderated_chain = SequentialChain( chains=[assistant_chain, moderator_chain], input_variables=[\"sentiment\", \"customer_request\"], verbose=True, ) # We can now run the chain. assistant_moderated_chain.run({\"sentiment\": \"rude\", \"customer_request\": customer_request})","title":"LLMChain"},{"location":"llm/#optimizing-inference-on-large-language-models-with-nvidia-tensorrt-llm","text":"https://developer.nvidia.com/blog/optimizing-inference-on-llms-with-tensorrt-llm-now-publicly-available/ https://github.com/furyhawk/TensorRT-LLM","title":"Optimizing Inference on Large Language Models with NVIDIA TensorRT-LLM"},{"location":"machine_learning/","text":"machine_learning Neural Network Playground https://furyhawk.github.io/playground/ Residual connections When the network get too deep, to adjust the parameters for each function in the chain based on the error(loss) recorded on the output layer, each sucessful layers include more noise. The noise start to overwhelm gradient information. This is the vanishing gradients problem. Batch normalization The paper stated that Batch normalization operates by \"reducing internal covariate shift\". Helps with gradient propagation, allowing for deeper networks. # Because the output of the Conv2D layer gets normalized, the layer doesn't need its own bias vector. x = layers.Conv2D(32,3,use_bias)(x) # do not include activation x = layers.BatchNormalization()(x) x = layers.Activation(\"relu\")(x) # place activation after BatchNormalization layer Batch normalization and Fine tuning When fine-tuning a model that includes BatchNormalization layers, leave these layers frozen( trainable=False ). Otherwise they will keep updating their internal mean and variance, which can interfere with the very small updates applied to the surrounding Conv2D layers. Steps per Epoch Based on what you said it sounds like you need a larger batch_size, and of course there are implications with that which could impact the steps_per_epoch and number of epochs. To solve for jumping-around - A **larger batch size** will give you a better gradient and will help to prevent jumping around - You may also want to consider a smaller learning rate, or a learning rate scheduler (or decay) to allow the network to \"settle in\" as it trains Feature Extraction All of the models in timm have consistent mechanisms for obtaining various types of features from the model for tasks besides classification. Penultimate Layer Features (Pre-Classifier Features) The features from the penultimate model layer can be obtained in several ways without requiring model surgery (although feel free to do surgery). One must first decide if they want pooled or un-pooled features. Unpooled There are three ways to obtain unpooled features. Without modifying the network, one can call model.forward_features(input) on any model instead of the usual model(input) . This will bypass the head classifier and global pooling for networks. If one wants to explicitly modify the network to return unpooled features, they can either create the model without a classifier and pooling, or remove it later. Both paths remove the parameters associated with the classifier from the network. forward_features() import torch import timm m = timm.create_model('xception41', pretrained=True) o = m(torch.randn(2, 3, 299, 299)) print(f'Original shape: {o.shape}') o = m.forward_features(torch.randn(2, 3, 299, 299)) print(f'Unpooled shape: {o.shape}') Output: Original shape: torch.Size([2, 1000]) Unpooled shape: torch.Size([2, 2048, 10, 10]) Create with no classifier and pooling import torch import timm m = timm.create_model('resnet50', pretrained=True, num_classes=0, global_pool='') o = m(torch.randn(2, 3, 224, 224)) print(f'Unpooled shape: {o.shape}') Output: Unpooled shape: torch.Size([2, 2048, 7, 7]) Remove it later import torch import timm m = timm.create_model('densenet121', pretrained=True) o = m(torch.randn(2, 3, 224, 224)) print(f'Original shape: {o.shape}') m.reset_classifier(0, '') o = m(torch.randn(2, 3, 224, 224)) print(f'Unpooled shape: {o.shape}') Output: Original shape: torch.Size([2, 1000]) Unpooled shape: torch.Size([2, 1024, 7, 7]) Pooled To modify the network to return pooled features, one can use forward_features() and pool/flatten the result themselves, or modify the network like above but keep pooling intact. Create with no classifier import torch import timm m = timm.create_model('resnet50', pretrained=True, num_classes=0) o = m(torch.randn(2, 3, 224, 224)) print(f'Pooled shape: {o.shape}') Output: Pooled shape: torch.Size([2, 2048]) Remove it later import torch import timm m = timm.create_model('ese_vovnet19b_dw', pretrained=True) o = m(torch.randn(2, 3, 224, 224)) print(f'Original shape: {o.shape}') m.reset_classifier(0) o = m(torch.randn(2, 3, 224, 224)) print(f'Pooled shape: {o.shape}') Output: Original shape: torch.Size([2, 1000]) Pooled shape: torch.Size([2, 1024]) Multi-scale Feature Maps (Feature Pyramid) Object detection, segmentation, keypoint, and a variety of dense pixel tasks require access to feature maps from the backbone network at multiple scales. This is often done by modifying the original classification network. Since each network varies quite a bit in structure, it's not uncommon to see only a few backbones supported in any given obj detection or segmentation library. timm allows a consistent interface for creating any of the included models as feature backbones that output feature maps for selected levels. A feature backbone can be created by adding the argument features_only=True to any create_model call. By default 5 strides will be output from most models (not all have that many), with the first starting at 2 (some start at 1 or 4). Create a feature map extraction model import torch import timm m = timm.create_model('resnest26d', features_only=True, pretrained=True) o = m(torch.randn(2, 3, 224, 224)) for x in o: print(x.shape) Output: torch.Size([2, 64, 112, 112]) torch.Size([2, 256, 56, 56]) torch.Size([2, 512, 28, 28]) torch.Size([2, 1024, 14, 14]) torch.Size([2, 2048, 7, 7]) Query the feature information After a feature backbone has been created, it can be queried to provide channel or resolution reduction information to the downstream heads without requiring static config or hardcoded constants. The .feature_info attribute is a class encapsulating the information about the feature extraction points. import torch import timm m = timm.create_model('regnety_032', features_only=True, pretrained=True) print(f'Feature channels: {m.feature_info.channels()}') o = m(torch.randn(2, 3, 224, 224)) for x in o: print(x.shape) Output: Feature channels: [32, 72, 216, 576, 1512] torch.Size([2, 32, 112, 112]) torch.Size([2, 72, 56, 56]) torch.Size([2, 216, 28, 28]) torch.Size([2, 576, 14, 14]) torch.Size([2, 1512, 7, 7]) Select specific feature levels or limit the stride There are two additional creation arguments impacting the output features. out_indices selects which indices to output output_stride limits the feature output stride of the network (also works in classification mode BTW) out_indices is supported by all models, but not all models have the same index to feature stride mapping. Look at the code or check feature_info to compare. The out indices generally correspond to the C(i+1)th feature level (a 2^(i+1) reduction). For most models, index 0 is the stride 2 features, and index 4 is stride 32. output_stride is achieved by converting layers to use dilated convolutions. Doing so is not always straightforward, some networks only support output_stride=32 . import torch import timm m = timm.create_model('ecaresnet101d', features_only=True, output_stride=8, out_indices=(2, 4), pretrained=True) print(f'Feature channels: {m.feature_info.channels()}') print(f'Feature reduction: {m.feature_info.reduction()}') o = m(torch.randn(2, 3, 320, 320)) for x in o: print(x.shape) Output: Feature channels: [512, 2048] Feature reduction: [8, 8] torch.Size([2, 512, 40, 40]) torch.Size([2, 2048, 40, 40]) GloabalPooling Reduce computation by 75%. Summarise features. GlobalMaxPooling Feature extraction after covn layer. Use of GlobalAvgPooling One advantage of global average pooling over the fully connected layers is that it is more native to the convolution structure by enforcing correspondences between feature maps and categories. Thus the feature maps can be easily interpreted as categories confidence maps. Another advantage is that there is no parameter to optimize in the global average pooling thus overfitting is avoided at this layer. Futhermore, global average pooling sums out the spatial information, thus it is more robust to spatial translations of the input. We can see global average pooling as a structural regularizer that explicitly enforces feature maps to be confidence maps of concepts (categories). This is made possible by the mlpconv layers, as they makes better approximation to the confidence maps than GLMs. How filter init Note that we use the same weight initialization formula as with the MLP. Weights are sampled randomly from a uniform distribution in the range [-1/fan-in, 1/fan-in], where fan-in is the number of inputs to a hidden unit. For MLPs, this was the number of units in the layer below. For CNNs however, we have to take into account the number of input feature maps and the size of the receptive fields. Transfer Learning In practice, very few people train an entire Convolutional Network from scratch (with random initialization), because it is relatively rare to have a dataset of sufficient size. Instead, it is common to pretrain a ConvNet on a very large dataset (e.g. ImageNet, which contains 1.2 million images with 1000 categories), and then use the ConvNet either as an initialization or a fixed feature extractor for the task of interest. The three major Transfer Learning scenarios look as follows: ConvNet as fixed feature extractor. Take a ConvNet pretrained on ImageNet, remove the last fully-connected layer (this layer's outputs are the 1000 class scores for a different task like ImageNet), then treat the rest of the ConvNet as a fixed feature extractor for the new dataset. In an AlexNet, this would compute a 4096-D vector for every image that contains the activations of the hidden layer immediately before the classifier. We call these features CNN codes. It is important for performance that these codes are ReLUd (i.e. thresholded at zero) if they were also thresholded during the training of the ConvNet on ImageNet (as is usually the case). Once you extract the 4096-D codes for all images, train a linear classifier (e.g. Linear SVM or Softmax classifier) for the new dataset. Fine-tuning the ConvNet. The second strategy is to not only replace and retrain the classifier on top of the ConvNet on the new dataset, but to also fine-tune the weights of the pretrained network by continuing the backpropagation. It is possible to fine-tune all the layers of the ConvNet, or it's possible to keep some of the earlier layers fixed (due to overfitting concerns) and only fine-tune some higher-level portion of the network. This is motivated by the observation that the earlier features of a ConvNet contain more generic features (e.g. edge detectors or color blob detectors) that should be useful to many tasks, but later layers of the ConvNet becomes progressively more specific to the details of the classes contained in the original dataset. In case of ImageNet for example, which contains many dog breeds, a significant portion of the representational power of the ConvNet may be devoted to features that are specific to differentiating between dog breeds. When and how to fine-tune? How do you decide what type of transfer learning you should perform on a new dataset? This is a function of several factors, but the two most important ones are the size of the new dataset (small or big), and its similarity to the original dataset (e.g. ImageNet-like in terms of the content of images and the classes, or very different, such as microscope images). Keeping in mind that ConvNet features are more generic in early layers and more original-dataset-specific in later layers, here are some common rules of thumb for navigating the 4 major scenarios: New dataset is small and similar to original dataset. Since the data is small, it is not a good idea to fine-tune the ConvNet due to overfitting concerns. Since the data is similar to the original data, we expect higher-level features in the ConvNet to be relevant to this dataset as well. Hence, the best idea might be to train a linear classifier on the CNN codes. New dataset is large and similar to the original dataset. Since we have more data, we can have more confidence that we won't overfit if we were to try to fine-tune through the full network. New dataset is small but very different from the original dataset. Since the data is small, it is likely best to only train a linear classifier. Since the dataset is very different, it might not be best to train the classifier form the top of the network, which contains more dataset-specific features. Instead, it might work better to train the SVM classifier from activations somewhere earlier in the network. New dataset is large and very different from the original dataset. Since the dataset is very large, we may expect that we can afford to train a ConvNet from scratch. However, in practice it is very often still beneficial to initialize with weights from a pretrained model. In this case, we would have enough data and confidence to fine-tune through the entire network. Practical advice. There are a few additional things to keep in mind when performing Transfer Learning: Constraints from pretrained models . Note that if you wish to use a pretrained network, you may be slightly constrained in terms of the architecture you can use for your new dataset. For example, you can't arbitrarily take out Conv layers from the pretrained network. However, some changes are straight-forward: Due to parameter sharing, you can easily run a pretrained network on images of different spatial size. This is clearly evident in the case of Conv/Pool layers because their forward function is independent of the input volume spatial size (as long as the strides \"fit\"). In case of FC layers, this still holds true because FC layers can be converted to a Convolutional Layer: For example, in an AlexNet, the final pooling volume before the first FC layer is of size [6x6x512]. Therefore, the FC layer looking at this volume is equivalent to having a Convolutional Layer that has receptive field size 6x6, and is applied with padding of 0. Learning rates . It's common to use a smaller learning rate for ConvNet weights that are being fine-tuned, in comparison to the (randomly-initialized) weights for the new linear classifier that computes the class scores of your new dataset. This is because we expect that the ConvNet weights are relatively good, so we don't wish to distort them too quickly and too much (especially while the new Linear Classifier above them is being trained from random initialization). Implications of a larger batch-size - Too large of a batch_size can produce memory problems, especially if you are using a GPU. Once you exceed the limit, dial it back until it works. This will help you find the max batch-size that your system can work with. - Too large of a batch size can get you stuck in a local minima, so if your training get stuck, I would reduce it some. Imagine here you are over-correcting the jumping-around and it's not jumping around enough to further minimize the loss function. When to reduce epochs - If your train error is very low, yet your test/validation is very high, then you have over-fit the model with too many epochs. - The best way to find the right balance is to use early-stopping with a validation test set. Here you can specify when to stop training, and save the weights for the network that gives you the best validation loss. (I highly recommend using this always) When to adjust steps-per-epoch - Traditionally, the steps per epoch is calculated as train_length // batch_size, since this will use all of the data points, one batch size worth at a time. - If you are augmenting the data, then you can stretch this a tad (sometimes I multiply that function above by 2 or 3 etc. But, if it's already training for too long, then I would just stick with the traditional approach. When to Scale Rule of thumb I follow here is any algorithm that computes distance or assumes normality, scale your features!!! Some examples of algorithms where feature scaling matters are: - k-nearest neighbors with an Euclidean distance measure is sensitive to magnitudes and hence should be scaled for all features to weigh in equally. - Scaling is critical, while performing Principal Component Analysis(PCA). PCA tries to get the features with maximum variance and the variance is high for high magnitude features. This skews the PCA towards high magnitude features. - We can speed up gradient descent by scaling. This is because \u03b8 will descend quickly on small ranges and slowly on large ranges, and so will oscillate inefficiently down to the optimum when the variables are very uneven. - Tree based models are not distance based models and can handle varying ranges of features. Hence, Scaling is not required while modelling trees. Algorithms like Linear Discriminant Analysis(LDA), Naive Bayes are by design equipped to handle this and gives weights to the features accordingly. Performing a features scaling in these algorithms may not have much effect. PyTorch Loss Function Cheatsheet PyTorch Loss-Input Confusion (Cheatsheet) torch.nn.functional.binary_cross_entropy takes logistic sigmoid values as inputs torch.nn.functional.binary_cross_entropy_with_logits takes logits as inputs torch.nn.functional.cross_entropy takes logits as inputs (performs log_softmax internally) torch.nn.functional.nll_loss is like cross_entropy but takes log-probabilities (log-softmax) values as inputs","title":"machine_learning"},{"location":"machine_learning/#machine_learning","text":"","title":"machine_learning"},{"location":"machine_learning/#neural-network-playground","text":"https://furyhawk.github.io/playground/","title":"Neural Network Playground"},{"location":"machine_learning/#residual-connections","text":"When the network get too deep, to adjust the parameters for each function in the chain based on the error(loss) recorded on the output layer, each sucessful layers include more noise. The noise start to overwhelm gradient information. This is the vanishing gradients problem.","title":"Residual connections"},{"location":"machine_learning/#batch-normalization","text":"The paper stated that Batch normalization operates by \"reducing internal covariate shift\". Helps with gradient propagation, allowing for deeper networks. # Because the output of the Conv2D layer gets normalized, the layer doesn't need its own bias vector. x = layers.Conv2D(32,3,use_bias)(x) # do not include activation x = layers.BatchNormalization()(x) x = layers.Activation(\"relu\")(x) # place activation after BatchNormalization layer","title":"Batch normalization"},{"location":"machine_learning/#batch-normalization-and-fine-tuning","text":"When fine-tuning a model that includes BatchNormalization layers, leave these layers frozen( trainable=False ). Otherwise they will keep updating their internal mean and variance, which can interfere with the very small updates applied to the surrounding Conv2D layers.","title":"Batch normalization and Fine tuning"},{"location":"machine_learning/#steps-per-epoch","text":"Based on what you said it sounds like you need a larger batch_size, and of course there are implications with that which could impact the steps_per_epoch and number of epochs. To solve for jumping-around - A **larger batch size** will give you a better gradient and will help to prevent jumping around - You may also want to consider a smaller learning rate, or a learning rate scheduler (or decay) to allow the network to \"settle in\" as it trains","title":"Steps per Epoch"},{"location":"machine_learning/#feature-extraction","text":"All of the models in timm have consistent mechanisms for obtaining various types of features from the model for tasks besides classification.","title":"Feature Extraction"},{"location":"machine_learning/#penultimate-layer-features-pre-classifier-features","text":"The features from the penultimate model layer can be obtained in several ways without requiring model surgery (although feel free to do surgery). One must first decide if they want pooled or un-pooled features.","title":"Penultimate Layer Features (Pre-Classifier Features)"},{"location":"machine_learning/#unpooled","text":"There are three ways to obtain unpooled features. Without modifying the network, one can call model.forward_features(input) on any model instead of the usual model(input) . This will bypass the head classifier and global pooling for networks. If one wants to explicitly modify the network to return unpooled features, they can either create the model without a classifier and pooling, or remove it later. Both paths remove the parameters associated with the classifier from the network.","title":"Unpooled"},{"location":"machine_learning/#forward_features","text":"import torch import timm m = timm.create_model('xception41', pretrained=True) o = m(torch.randn(2, 3, 299, 299)) print(f'Original shape: {o.shape}') o = m.forward_features(torch.randn(2, 3, 299, 299)) print(f'Unpooled shape: {o.shape}') Output: Original shape: torch.Size([2, 1000]) Unpooled shape: torch.Size([2, 2048, 10, 10])","title":"forward_features()"},{"location":"machine_learning/#create-with-no-classifier-and-pooling","text":"import torch import timm m = timm.create_model('resnet50', pretrained=True, num_classes=0, global_pool='') o = m(torch.randn(2, 3, 224, 224)) print(f'Unpooled shape: {o.shape}') Output: Unpooled shape: torch.Size([2, 2048, 7, 7])","title":"Create with no classifier and pooling"},{"location":"machine_learning/#remove-it-later","text":"import torch import timm m = timm.create_model('densenet121', pretrained=True) o = m(torch.randn(2, 3, 224, 224)) print(f'Original shape: {o.shape}') m.reset_classifier(0, '') o = m(torch.randn(2, 3, 224, 224)) print(f'Unpooled shape: {o.shape}') Output: Original shape: torch.Size([2, 1000]) Unpooled shape: torch.Size([2, 1024, 7, 7])","title":"Remove it later"},{"location":"machine_learning/#pooled","text":"To modify the network to return pooled features, one can use forward_features() and pool/flatten the result themselves, or modify the network like above but keep pooling intact.","title":"Pooled"},{"location":"machine_learning/#create-with-no-classifier","text":"import torch import timm m = timm.create_model('resnet50', pretrained=True, num_classes=0) o = m(torch.randn(2, 3, 224, 224)) print(f'Pooled shape: {o.shape}') Output: Pooled shape: torch.Size([2, 2048])","title":"Create with no classifier"},{"location":"machine_learning/#remove-it-later_1","text":"import torch import timm m = timm.create_model('ese_vovnet19b_dw', pretrained=True) o = m(torch.randn(2, 3, 224, 224)) print(f'Original shape: {o.shape}') m.reset_classifier(0) o = m(torch.randn(2, 3, 224, 224)) print(f'Pooled shape: {o.shape}') Output: Original shape: torch.Size([2, 1000]) Pooled shape: torch.Size([2, 1024])","title":"Remove it later"},{"location":"machine_learning/#multi-scale-feature-maps-feature-pyramid","text":"Object detection, segmentation, keypoint, and a variety of dense pixel tasks require access to feature maps from the backbone network at multiple scales. This is often done by modifying the original classification network. Since each network varies quite a bit in structure, it's not uncommon to see only a few backbones supported in any given obj detection or segmentation library. timm allows a consistent interface for creating any of the included models as feature backbones that output feature maps for selected levels. A feature backbone can be created by adding the argument features_only=True to any create_model call. By default 5 strides will be output from most models (not all have that many), with the first starting at 2 (some start at 1 or 4).","title":"Multi-scale Feature Maps (Feature Pyramid)"},{"location":"machine_learning/#create-a-feature-map-extraction-model","text":"import torch import timm m = timm.create_model('resnest26d', features_only=True, pretrained=True) o = m(torch.randn(2, 3, 224, 224)) for x in o: print(x.shape) Output: torch.Size([2, 64, 112, 112]) torch.Size([2, 256, 56, 56]) torch.Size([2, 512, 28, 28]) torch.Size([2, 1024, 14, 14]) torch.Size([2, 2048, 7, 7])","title":"Create a feature map extraction model"},{"location":"machine_learning/#query-the-feature-information","text":"After a feature backbone has been created, it can be queried to provide channel or resolution reduction information to the downstream heads without requiring static config or hardcoded constants. The .feature_info attribute is a class encapsulating the information about the feature extraction points. import torch import timm m = timm.create_model('regnety_032', features_only=True, pretrained=True) print(f'Feature channels: {m.feature_info.channels()}') o = m(torch.randn(2, 3, 224, 224)) for x in o: print(x.shape) Output: Feature channels: [32, 72, 216, 576, 1512] torch.Size([2, 32, 112, 112]) torch.Size([2, 72, 56, 56]) torch.Size([2, 216, 28, 28]) torch.Size([2, 576, 14, 14]) torch.Size([2, 1512, 7, 7])","title":"Query the feature information"},{"location":"machine_learning/#select-specific-feature-levels-or-limit-the-stride","text":"There are two additional creation arguments impacting the output features. out_indices selects which indices to output output_stride limits the feature output stride of the network (also works in classification mode BTW) out_indices is supported by all models, but not all models have the same index to feature stride mapping. Look at the code or check feature_info to compare. The out indices generally correspond to the C(i+1)th feature level (a 2^(i+1) reduction). For most models, index 0 is the stride 2 features, and index 4 is stride 32. output_stride is achieved by converting layers to use dilated convolutions. Doing so is not always straightforward, some networks only support output_stride=32 . import torch import timm m = timm.create_model('ecaresnet101d', features_only=True, output_stride=8, out_indices=(2, 4), pretrained=True) print(f'Feature channels: {m.feature_info.channels()}') print(f'Feature reduction: {m.feature_info.reduction()}') o = m(torch.randn(2, 3, 320, 320)) for x in o: print(x.shape) Output: Feature channels: [512, 2048] Feature reduction: [8, 8] torch.Size([2, 512, 40, 40]) torch.Size([2, 2048, 40, 40])","title":"Select specific feature levels or limit the stride"},{"location":"machine_learning/#gloabalpooling","text":"Reduce computation by 75%. Summarise features.","title":"GloabalPooling"},{"location":"machine_learning/#globalmaxpooling","text":"Feature extraction after covn layer.","title":"GlobalMaxPooling"},{"location":"machine_learning/#use-of-globalavgpooling","text":"One advantage of global average pooling over the fully connected layers is that it is more native to the convolution structure by enforcing correspondences between feature maps and categories. Thus the feature maps can be easily interpreted as categories confidence maps. Another advantage is that there is no parameter to optimize in the global average pooling thus overfitting is avoided at this layer. Futhermore, global average pooling sums out the spatial information, thus it is more robust to spatial translations of the input. We can see global average pooling as a structural regularizer that explicitly enforces feature maps to be confidence maps of concepts (categories). This is made possible by the mlpconv layers, as they makes better approximation to the confidence maps than GLMs.","title":"Use of GlobalAvgPooling"},{"location":"machine_learning/#how-filter-init","text":"Note that we use the same weight initialization formula as with the MLP. Weights are sampled randomly from a uniform distribution in the range [-1/fan-in, 1/fan-in], where fan-in is the number of inputs to a hidden unit. For MLPs, this was the number of units in the layer below. For CNNs however, we have to take into account the number of input feature maps and the size of the receptive fields.","title":"How filter init"},{"location":"machine_learning/#transfer-learning","text":"In practice, very few people train an entire Convolutional Network from scratch (with random initialization), because it is relatively rare to have a dataset of sufficient size. Instead, it is common to pretrain a ConvNet on a very large dataset (e.g. ImageNet, which contains 1.2 million images with 1000 categories), and then use the ConvNet either as an initialization or a fixed feature extractor for the task of interest. The three major Transfer Learning scenarios look as follows: ConvNet as fixed feature extractor. Take a ConvNet pretrained on ImageNet, remove the last fully-connected layer (this layer's outputs are the 1000 class scores for a different task like ImageNet), then treat the rest of the ConvNet as a fixed feature extractor for the new dataset. In an AlexNet, this would compute a 4096-D vector for every image that contains the activations of the hidden layer immediately before the classifier. We call these features CNN codes. It is important for performance that these codes are ReLUd (i.e. thresholded at zero) if they were also thresholded during the training of the ConvNet on ImageNet (as is usually the case). Once you extract the 4096-D codes for all images, train a linear classifier (e.g. Linear SVM or Softmax classifier) for the new dataset. Fine-tuning the ConvNet. The second strategy is to not only replace and retrain the classifier on top of the ConvNet on the new dataset, but to also fine-tune the weights of the pretrained network by continuing the backpropagation. It is possible to fine-tune all the layers of the ConvNet, or it's possible to keep some of the earlier layers fixed (due to overfitting concerns) and only fine-tune some higher-level portion of the network. This is motivated by the observation that the earlier features of a ConvNet contain more generic features (e.g. edge detectors or color blob detectors) that should be useful to many tasks, but later layers of the ConvNet becomes progressively more specific to the details of the classes contained in the original dataset. In case of ImageNet for example, which contains many dog breeds, a significant portion of the representational power of the ConvNet may be devoted to features that are specific to differentiating between dog breeds.","title":"Transfer Learning"},{"location":"machine_learning/#when-and-how-to-fine-tune","text":"How do you decide what type of transfer learning you should perform on a new dataset? This is a function of several factors, but the two most important ones are the size of the new dataset (small or big), and its similarity to the original dataset (e.g. ImageNet-like in terms of the content of images and the classes, or very different, such as microscope images). Keeping in mind that ConvNet features are more generic in early layers and more original-dataset-specific in later layers, here are some common rules of thumb for navigating the 4 major scenarios: New dataset is small and similar to original dataset. Since the data is small, it is not a good idea to fine-tune the ConvNet due to overfitting concerns. Since the data is similar to the original data, we expect higher-level features in the ConvNet to be relevant to this dataset as well. Hence, the best idea might be to train a linear classifier on the CNN codes. New dataset is large and similar to the original dataset. Since we have more data, we can have more confidence that we won't overfit if we were to try to fine-tune through the full network. New dataset is small but very different from the original dataset. Since the data is small, it is likely best to only train a linear classifier. Since the dataset is very different, it might not be best to train the classifier form the top of the network, which contains more dataset-specific features. Instead, it might work better to train the SVM classifier from activations somewhere earlier in the network. New dataset is large and very different from the original dataset. Since the dataset is very large, we may expect that we can afford to train a ConvNet from scratch. However, in practice it is very often still beneficial to initialize with weights from a pretrained model. In this case, we would have enough data and confidence to fine-tune through the entire network.","title":"When and how to fine-tune?"},{"location":"machine_learning/#practical-advice","text":"There are a few additional things to keep in mind when performing Transfer Learning: Constraints from pretrained models . Note that if you wish to use a pretrained network, you may be slightly constrained in terms of the architecture you can use for your new dataset. For example, you can't arbitrarily take out Conv layers from the pretrained network. However, some changes are straight-forward: Due to parameter sharing, you can easily run a pretrained network on images of different spatial size. This is clearly evident in the case of Conv/Pool layers because their forward function is independent of the input volume spatial size (as long as the strides \"fit\"). In case of FC layers, this still holds true because FC layers can be converted to a Convolutional Layer: For example, in an AlexNet, the final pooling volume before the first FC layer is of size [6x6x512]. Therefore, the FC layer looking at this volume is equivalent to having a Convolutional Layer that has receptive field size 6x6, and is applied with padding of 0. Learning rates . It's common to use a smaller learning rate for ConvNet weights that are being fine-tuned, in comparison to the (randomly-initialized) weights for the new linear classifier that computes the class scores of your new dataset. This is because we expect that the ConvNet weights are relatively good, so we don't wish to distort them too quickly and too much (especially while the new Linear Classifier above them is being trained from random initialization). Implications of a larger batch-size - Too large of a batch_size can produce memory problems, especially if you are using a GPU. Once you exceed the limit, dial it back until it works. This will help you find the max batch-size that your system can work with. - Too large of a batch size can get you stuck in a local minima, so if your training get stuck, I would reduce it some. Imagine here you are over-correcting the jumping-around and it's not jumping around enough to further minimize the loss function. When to reduce epochs - If your train error is very low, yet your test/validation is very high, then you have over-fit the model with too many epochs. - The best way to find the right balance is to use early-stopping with a validation test set. Here you can specify when to stop training, and save the weights for the network that gives you the best validation loss. (I highly recommend using this always) When to adjust steps-per-epoch - Traditionally, the steps per epoch is calculated as train_length // batch_size, since this will use all of the data points, one batch size worth at a time. - If you are augmenting the data, then you can stretch this a tad (sometimes I multiply that function above by 2 or 3 etc. But, if it's already training for too long, then I would just stick with the traditional approach.","title":"Practical advice."},{"location":"machine_learning/#when-to-scale","text":"Rule of thumb I follow here is any algorithm that computes distance or assumes normality, scale your features!!! Some examples of algorithms where feature scaling matters are: - k-nearest neighbors with an Euclidean distance measure is sensitive to magnitudes and hence should be scaled for all features to weigh in equally. - Scaling is critical, while performing Principal Component Analysis(PCA). PCA tries to get the features with maximum variance and the variance is high for high magnitude features. This skews the PCA towards high magnitude features. - We can speed up gradient descent by scaling. This is because \u03b8 will descend quickly on small ranges and slowly on large ranges, and so will oscillate inefficiently down to the optimum when the variables are very uneven. - Tree based models are not distance based models and can handle varying ranges of features. Hence, Scaling is not required while modelling trees. Algorithms like Linear Discriminant Analysis(LDA), Naive Bayes are by design equipped to handle this and gives weights to the features accordingly. Performing a features scaling in these algorithms may not have much effect.","title":"When to Scale"},{"location":"machine_learning/#pytorch-loss-function-cheatsheet","text":"PyTorch Loss-Input Confusion (Cheatsheet) torch.nn.functional.binary_cross_entropy takes logistic sigmoid values as inputs torch.nn.functional.binary_cross_entropy_with_logits takes logits as inputs torch.nn.functional.cross_entropy takes logits as inputs (performs log_softmax internally) torch.nn.functional.nll_loss is like cross_entropy but takes log-probabilities (log-softmax) values as inputs","title":"PyTorch Loss Function Cheatsheet"},{"location":"macos/","text":"macos find . -name ._\\* -delete","title":"macos"},{"location":"macos/#macos","text":"find . -name ._\\* -delete","title":"macos"},{"location":"mkdocs/","text":"Welcome to MkDocs For full documentation visit mkdocs.org . Commands mkdocs new [dir-name] - Create a new project. mkdocs serve - Start the live-reloading docs server. mkdocs build - Build the documentation site. mkdocs -h - Print help message and exit. mkdocs gh-deploy - Deploy to github branch gh-pages . Deploying Your Docs - MkDocs Project layout mkdocs.yml # The configuration file. docs/ index.md # The documentation homepage. ... # Other markdown pages, images and other files.","title":"MkDocs"},{"location":"mkdocs/#welcome-to-mkdocs","text":"For full documentation visit mkdocs.org .","title":"Welcome to MkDocs"},{"location":"mkdocs/#commands","text":"mkdocs new [dir-name] - Create a new project. mkdocs serve - Start the live-reloading docs server. mkdocs build - Build the documentation site. mkdocs -h - Print help message and exit. mkdocs gh-deploy - Deploy to github branch gh-pages . Deploying Your Docs - MkDocs","title":"Commands"},{"location":"mkdocs/#project-layout","text":"mkdocs.yml # The configuration file. docs/ index.md # The documentation homepage. ... # Other markdown pages, images and other files.","title":"Project layout"},{"location":"mqtt/","text":"MQTT https://github.com/furyhawk/scratchpad/tree/main/mtqq RabbitMQ # latest RabbitMQ 3.12 docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.12-management HiveMQ docker run --name hivemq-edge -d -p 1883:1883 -p 8080:8080 hivemq/hivemq-edge How to setup Mosquitto MQTT Broker using docker These instructions will work on any Debian based OS including Ubuntu, RaspberryPi, WSL2 etc... (For non-Debian distros, commands for installation need to be tweaked) By default the config allows only to use local connections for security reasons but since authentication is enabled below, that's not the case. 1. Install docker Latest instructions are here on docker website. You can also use this script - install-docker.sh 2. Create base folder for mqtt configuration mkdir mqtt5 cd mqtt5 # for storing mosquitto.conf and pwfile (for password) mkdir config 3. Create Mosquitto config file - mosquitto.conf nano config/mosquitto.conf Basic configuration file content below including websocket config allow_anonymous false listener 1883 listener 9001 protocol websockets persistence true password_file /mosquitto/config/pwfile persistence_file mosquitto.db persistence_location /mosquitto/data/ 4. Create Mosquitto password file - pwfile touch config/pwfile 5. Create docker-compose file called 'docker-compose.yml' version: \"3.7\" services: # mqtt5 eclipse-mosquitto mqtt5: image: eclipse-mosquitto container_name: mqtt5 ports: - \"1883:1883\" #default mqtt port - \"9001:9001\" #default mqtt port for websockets volumes: - ./config:/mosquitto/config:rw - ./data:/mosquitto/data:rw - ./log:/mosquitto/log:rw restart: unless-stopped # volumes for mapping data,config and log volumes: config: data: log: networks: default: name: mqtt5-network 6. Create and run docker container for MQTT # In case you don't have docker-compose you can install it # sudo apt install docker-compose # Run the docker container for mqtt docker compose -p mqtt5 up -d Check if the container is up and working (note down container-id) docker ps 7. Create a user/password in the pwfile # login interactively into the mqtt container docker exec -it sh # add user and it will prompt for password mosquitto_passwd -c /mosquitto/config/pwfile user1 # delete user command format mosquitto_passwd -D /mosquitto/config/pwfile # type 'exit' to exit out of docker container prompt Then restart the container docker restart 8. Time to test !!! Install mosquitto client tools for testing sudo apt install mosquitto-clients Let us start Subscriber now - topic name => 'hello/topic' # Without authentication mosquitto_sub -v -t 'hello/topic' # With authentication mosquitto_sub -v -t 'hello/topic' -u user1 -P # Alternate way in url format # Format => mqtt(s)://[username[:password]@]host[:port]/topic mosquitto_sub -v -L mqtt://user1:abc123@localhost/test/topic Let us start Publising to that topic # Without authentication mosquitto_pub -t 'hello/topic' -m 'hello MQTT' # With authentication mosquitto_pub -t 'hello/topic' -m 'hello MQTT' -u user1 -P # Alternate way in url format # Format => mqtt(s)://[username[:password]@]host[:port]/topic mosquitto_pub -L mqtt://user1:abc123@localhost/test/topic -m 'hello MQTT' You can find C/C++ code for mosquitto client Check main.cpp for the mosquitto client code. You can also install a nice MQTT Web Client Read more about it here => https://mqttx.app/ sudo docker run -d --name mqttx-web -p 80:80 emqx/mqttx-web Source/Reference for Mosquitto Github => https://github.com/eclipse/mosquitto","title":"mqtt"},{"location":"mqtt/#mqtt","text":"https://github.com/furyhawk/scratchpad/tree/main/mtqq","title":"MQTT"},{"location":"mqtt/#rabbitmq","text":"# latest RabbitMQ 3.12 docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.12-management","title":"RabbitMQ"},{"location":"mqtt/#hivemq","text":"docker run --name hivemq-edge -d -p 1883:1883 -p 8080:8080 hivemq/hivemq-edge","title":"HiveMQ"},{"location":"mqtt/#how-to-setup-mosquitto-mqtt-broker-using-docker","text":"These instructions will work on any Debian based OS including Ubuntu, RaspberryPi, WSL2 etc... (For non-Debian distros, commands for installation need to be tweaked) By default the config allows only to use local connections for security reasons but since authentication is enabled below, that's not the case.","title":"How to setup Mosquitto MQTT Broker using docker"},{"location":"mqtt/#1-install-docker","text":"Latest instructions are here on docker website. You can also use this script - install-docker.sh","title":"1. Install docker"},{"location":"mqtt/#2-create-base-folder-for-mqtt-configuration","text":"mkdir mqtt5 cd mqtt5 # for storing mosquitto.conf and pwfile (for password) mkdir config","title":"2. Create base folder for mqtt configuration"},{"location":"mqtt/#3-create-mosquitto-config-file-mosquittoconf","text":"nano config/mosquitto.conf Basic configuration file content below including websocket config allow_anonymous false listener 1883 listener 9001 protocol websockets persistence true password_file /mosquitto/config/pwfile persistence_file mosquitto.db persistence_location /mosquitto/data/","title":"3. Create Mosquitto config file - mosquitto.conf"},{"location":"mqtt/#4-create-mosquitto-password-file-pwfile","text":"touch config/pwfile","title":"4. Create Mosquitto password file - pwfile"},{"location":"mqtt/#5-create-docker-compose-file-called-docker-composeyml","text":"version: \"3.7\" services: # mqtt5 eclipse-mosquitto mqtt5: image: eclipse-mosquitto container_name: mqtt5 ports: - \"1883:1883\" #default mqtt port - \"9001:9001\" #default mqtt port for websockets volumes: - ./config:/mosquitto/config:rw - ./data:/mosquitto/data:rw - ./log:/mosquitto/log:rw restart: unless-stopped # volumes for mapping data,config and log volumes: config: data: log: networks: default: name: mqtt5-network","title":"5. Create docker-compose file called 'docker-compose.yml'"},{"location":"mqtt/#6-create-and-run-docker-container-for-mqtt","text":"# In case you don't have docker-compose you can install it # sudo apt install docker-compose # Run the docker container for mqtt docker compose -p mqtt5 up -d","title":"6. Create and run docker container for MQTT"},{"location":"mqtt/#check-if-the-container-is-up-and-working-note-down-container-id","text":"docker ps","title":"Check if the container is up and working (note down container-id)"},{"location":"mqtt/#7-create-a-userpassword-in-the-pwfile","text":"# login interactively into the mqtt container docker exec -it sh # add user and it will prompt for password mosquitto_passwd -c /mosquitto/config/pwfile user1 # delete user command format mosquitto_passwd -D /mosquitto/config/pwfile # type 'exit' to exit out of docker container prompt Then restart the container docker restart ","title":"7. Create a user/password in the pwfile"},{"location":"mqtt/#8-time-to-test","text":"","title":"8. Time to test !!!"},{"location":"mqtt/#install-mosquitto-client-tools-for-testing","text":"sudo apt install mosquitto-clients","title":"Install mosquitto client tools for testing"},{"location":"mqtt/#let-us-start-subscriber-now-topic-name-hellotopic","text":"# Without authentication mosquitto_sub -v -t 'hello/topic' # With authentication mosquitto_sub -v -t 'hello/topic' -u user1 -P # Alternate way in url format # Format => mqtt(s)://[username[:password]@]host[:port]/topic mosquitto_sub -v -L mqtt://user1:abc123@localhost/test/topic","title":"Let us start Subscriber now - topic name => 'hello/topic'"},{"location":"mqtt/#let-us-start-publising-to-that-topic","text":"# Without authentication mosquitto_pub -t 'hello/topic' -m 'hello MQTT' # With authentication mosquitto_pub -t 'hello/topic' -m 'hello MQTT' -u user1 -P # Alternate way in url format # Format => mqtt(s)://[username[:password]@]host[:port]/topic mosquitto_pub -L mqtt://user1:abc123@localhost/test/topic -m 'hello MQTT'","title":"Let us start Publising to that topic"},{"location":"mqtt/#you-can-find-cc-code-for-mosquitto-client","text":"Check main.cpp for the mosquitto client code.","title":"You can find C/C++ code for mosquitto client"},{"location":"mqtt/#you-can-also-install-a-nice-mqtt-web-client","text":"Read more about it here => https://mqttx.app/ sudo docker run -d --name mqttx-web -p 80:80 emqx/mqttx-web","title":"You can also install a nice MQTT Web Client"},{"location":"mqtt/#sourcereference-for-mosquitto","text":"Github => https://github.com/eclipse/mosquitto","title":"Source/Reference for Mosquitto"},{"location":"mypy/","text":"mypy Type check # mypy: ignore-errors # type: ignore Built-in types Simple types Here are examples of some common built-in types: Type Description int integer float floating point number bool boolean value (subclass of int) str text, sequence of unicode codepoints bytes 8-bit string, sequence of byte values object an arbitrary object (object is the common base class) All built-in classes can be used as types.","title":"mypy"},{"location":"mypy/#mypy-type-check","text":"# mypy: ignore-errors # type: ignore","title":"mypy Type check"},{"location":"mypy/#built-in-types","text":"","title":"Built-in types"},{"location":"mypy/#simple-types","text":"Here are examples of some common built-in types: Type Description int integer float floating point number bool boolean value (subclass of int) str text, sequence of unicode codepoints bytes 8-bit string, sequence of byte values object an arbitrary object (object is the common base class) All built-in classes can be used as types.","title":"Simple types"},{"location":"nes/","text":"nes faq Phantom Fighter ==================================================================== Phantom Fighter Complete FAQ/Walkthrough September 05, 2003 Version 2.00 author: Aaron Madrinan (snoocete) e-mail: snoocete@yahoo.com subject: \"RE: Phantom Fighter FAQ/Walkthrough\" ==================================================================== --- ## ---> INTRODUCTION The gaming world is changing very rapidly. Games are released on breakneck speed that we need more than a lifetime to play even just the good ones. The result? The other not-so-famous games are forgotten and put aside by future gamers. In other words, they become underrated. Such is what happened to a game called Phantom Fighter. I've played it since I was a kid. Though not as extremely fun as games like Kirby or Pokemon, it very much dwells on the concept in which Contra flourished: you can never let the game beat you. Supposedly, it would be as famous as that. But, heck there wasn't even a walkthrough on GameFAQs, only some codes that lets you skip to the last town and a very bad review. Sigh. You want my opinion? Here is what I can say: don't let the big names fool you. A lot of gems are hidden in the dump of really bad games, I admit, but they only need a little spotlight to shine. Here is my share of that spotlight =). --- ## ---> DISCLAIMER You are free to save this guide onto your hard drive and/or print for your personal viewing pleasure. You may distibute it to other people as long as you don't claim it as yours, okay? Since this game doesn't have that much coverage, you may post this at your site in its full, unedited version even without my permission as long as its credits point to me. Deal! --- ## ---> CONTENTS 1. Updates --- 2. FAQ --- 3. Walkthrough --- 4. Credits --- This is FAQ #2, hope I got a bit better. --- ## ---> UPDATES -2.00- -21Kb- Look above (^\\_^). It's a whole number change! A kind soul mailed to me the names of the Kyonshies. Yay! Added peace and order (where there was chaos). Formatting change. E-mail change. Lastly, look at the disclaimer. -1.00- -19Kb- This FAQ is complete, however the Japanese names of the Kyonshies are missing. Not that anybody would care, but it still important. If you have an instruction manual (because I kinda lost it), kindly e-mail me and let me know, ok? --- ## ---> FAQ --Who are you? Just a humble freelance (a bit newbie-ish, but wants to learn a lot) FAQ writer who writes for his favorite games whenever he has the time. --What are these \"Kyonshi\" that I'm fighting? If you have ever watched the anime series \"Shaman King\", or knew a bit about Chinese folkore, you have the idea. These are what we call the undead; they are supposed to be immobile, rotting bodies. They still are, but they are being controlled by some mysterious power. You know about rigor mortis? It also applies to these phantoms. That explains why they could only move their feet and knees to hop and turn around, and their elbows to puncture your face with their dirty, sharp nails. --How do I fight these enemies? You have to move around. A lot. Whenever you stay on one spot, the Kyonshi will follow you with small hops, then when it gets in range, it suddenly does a big dive toward you. Unless you hit the Kyonshi in a critical point or you kill it in transit, you cannot stop that from homing to you. So you got to move. That's your ace against these creatures. They have lockjaw and every known bone and muscle disease, while you could just move out of their way. For example, the moment they land on the ground, give them a kick or two on their sleeves then they will be knocked down. When they wake up and hops around again, repeat. Soon they will burst in flames, and the door that lets you continue through the building opens. --What does that man who follows me around do? That's your assistant. Whenever you are inside a building, if you return to the exit, you talk to him. He asks if you want to have an item, stay at the building, or just leave the place. --And those items are...? There are a total of four (4) items in the game. You can find them from the households you rescue from kyonshies. They replace the use of the punch attack, but it's worth it. 1. Tonten - the kyonshi you show this with is knocked back to the ground because a) there is holy white light coming from it, or b) he saw his face, squirming with maggots (hey, thanks mate for the mirror, 'bout time I fix my hair... what the he... _faints_). DON'T use this when you're about to get hit; it also acts as your armor, and because it's a mirror and was hit by diamond hard claws, you know what happens. 2. Sacred Sword - even a swordless Link (of Legend of Zelda) would not use this nicely-named weapon (he might even throw it back at you so don't even try). It does little damage, and it also gets broken easily as if it is made by glass. The only edge it has is that it has a rather long range for a physical attack, you could corner a kyonshi with it, but you would like to beat them up yourself, wouldn't you? 3. Talisman - freezes the enemy in place until it's effect is gone or they got hit. You have to make contact with the kyonshi you want to victimize it with, so there is a chance you get hit and the item breaks. Useful on the tall kyonshies, you could sneak under them, paste the talisman on their belly, then kick them in the butt. 4. Bell - If you use this in the middle of a heated battle, it does nothing except probably makes a little chime that is pleasant to hear for you but instead boils the blood of the kyonshi, making him more aggressive. Actually, use it to control the kyonshi kid. --What are the scrolls for? Whenever you cleanse a building from kyonshies, you get a varying amount of scrolls. You use them at the Training Hall to learn new abilities. --The guard won't let me in because I can't answer his questions... got a list there that could help me? Please? Of course I do. I intend to make this the complete-est FAQ you'll ever see (for PF, of course). Here they are (if you noticed, the stupid comments are gone. just learned not to learn from the likes of ArchNacho and Tortilla Godzilla): -Format- Q - Question A - Answer Q: Why do Kyonshies come out only at night? A: Hate the sun. Q: What country do Samurais come from? A: Japan Q: What is another word for one? A: Any answer will do. Q: Who built the Great Wall of China? A: Any answer will do. Q: What is the food Kyonshies hate? A: Ice Cream Q: Who created Ultima? A: Lord British SC: Who is he? Q: What is Kyonshi's most powerful weapon? A: Sharp claws Q: What is the skill when you hold an opponent's arms from the back and throw him backward? A: Dragon Suplex Q: What kind of place do Kyonshies usually live in? A: Any answer will do. Q: What is a horrible skill when you head-banged against an opponent's head? A: Any answer will do. Q: Name an FCI video game. A: Any answer will do. Q: What is the famous Chinese newspaper? A: Wall poster Q: How many stars are there in the American flag? A: 50 Q: What is the best thing used to capture Kyonshies? A: Urn Q: What is a Chinese martial art usually called? A: Kung fu Q: What is the best method to make sure Kyonshies never revive again? A: Fry in oil Q: What is the teaching taught by Confucius called? A: Confucianism Q: What's the name of George Bush's dog? A: Millie Q: Who is the emperor called the Last Emperor? A: Fugi --Can I kill the guard? If you are not a super nerd, who knows NES rom-editing to change the layer which the guard resides to your characters layer and set its attributes to \"hittable\", has an HP count, and many other things, definitely no. I know it's a big disappointment. --What are these new abilities you will learn from the Training Hall? New techniques in martial arts. When you started the game, you have a weak punch and a weak kick. Learn these and you will become stronger and better to fight kyonshies. Town 1 (2scr) =2 Thrust - punch and you will hit the enemy twice =2 Kick - same with 2 Thrust but you kick instead =High Jump - self-explanatory =Wolf Move - speed up Town 2 (6scr) =3 Thrust - three fists-of-fury in one swift delivery =Turn Kick - most versatile move in the game =Wind Jump - jump and hold left/right to do a spin jump =Tiger Move - more speed up =Mirage Move - combine with Tiger Mv. for better effect Town 3 (18scr) =Mirage Walk - move while crouching =Dragon Move - whoosh! last speed upgrade Town 4 (50scr) =Wind Kick - jump then kick furiously in mid-air to form a spin kick =Mirage Thrust - punch while crouching Town 5 = Town 6 (80scr) =Jump Kick - hold left/right then kick to see this fantastic move Town 7 (90scr) =4 Thrust - most powerful move in the game Town 8 = --What are the names of these Kyonshies? What I have here are unofficial names which I made up by their looks, color, and strength: Pink - weak household form of kyonshi. Green - stronger and faster than the pink Dwarf - a real pain, their size and speed make them hard to hit. Tall - vulnerable to punch attacks, mirage thrust Fat - looks tough but slow, easy target Short-armed - has a high resistance to hits, has high attack power Their original Japanese names (courtesy of SatelliteGeibor@aol.com): Pink Kyonshi: Sosekushi Green Kyonshi: Zanshi Fat Kyonshi: Kimenshi Dwarf Kyonshi: WeeKyonshi (literally) Tall Kyonshi: Ryukyoshi Graveyard Ghost: Shanshi --Know any GameGenie codes? Nope. Go to the Codes and Secrets page of this game and you might find something to help, you cheater! --- ## ---> WALKTHROUGH In case you don't have a manual, here is the juicy bit of the story: You are Kenchi, China's savior from the Kyonshi, the eastern counter- part of the vampire. Someone had let them loose on seven towns, with the eighth one being totally controlled by that being. Of course, you as the hero walks in the scene and puts a stop to the onslaught. However, it won't be easy: the Kyonshi are undead, has little sense of pain and their conscience all gone. Which makes them more fun to battle, isn't it? +---------------------------------+ |Town 1 - A taste of Kyonshi power| +---------------------------------+ Enter the temple. This holy building looks always the same in every town you come to; it also functions the same thing; as a rest point for your character. That's why you should also invade this first. Oh, the controls: left/right: moves you in the direction you press. up: jump (not so high right now, so don't use it much) down: crouch/duck B button: punch A button: kick Select: when outside, changes message display speed Start: pauses the game Kill the introductory kyonshi inside, then the temple could now be used as an inn. The ASCII art below shows the lay-out of the town: T S O S S O H S O I I B |_||_||_||_||_||_||_||_||_||_||_||_| Legend: T-Temple H-Training Hall I-Item inside S-Scroll inside O-Orb inside B-Boss inside Enter houses and fight kyonshies to have scrolls and some items. Pink kyonshies are weaker than green ones, but later in the game, most of the time you will fight green ones so you have to practice on them, too. Heal at the temple if you are hurt. Every time you get enough scrolls, head to the Training Hall and learn a new ability, in the order 2 Kick, Wolf Move, 2 Thrust, then High Jump. Afterwards, enter the Orb buildings, defeat the more difficult kyonshies there and get the Orb they leave. After you have three orbs, enter the boss cave and fight... AFTERIMAGE KYONSHI (Japanese name: Genyoshi) Difficulty: */ * --I just made up the name, sorry. This boss is small but jumps ultra high. Also it takes up many hits to bring down so don't you dare stand in its path to kick/punch. You could a) wait until it reaches the high point in its jump, then as it falls down, attack it, or b) my favorite way. Use the talisman to freeze it then kick it from behind. Rinse and repeat. Your reward? Being congratulated by your assistant and access to the next town =). Don't forget to copy the password. +-------------------------+ |Town 2 - The baby Kyonshi| +-------------------------+ I H S S O O T I O \\* I B |_||_||_||_||_||_||_||_||_||_||_||_| The \\* contains Conshi, the baby Kyonshi. You can have him in your party by letting your assistant be snatched by the light in the graveyard, defeating her (she's very strong, but not too hard), then afterwards you will get the bell. You may want to try it to know how it feels to be a Kyonshi; otherwise do the same modus operandi in Town 1 (learn the Turn Kick as soon as you can), and at the end is another cave... KNIFE KYONSHI (Japanese name: Raunshi) Difficulty: /*** --As soon as you enter, stop for the boss is throwing knives at you. After enough knives, he pauses, that's your signal to get near and whack him. If he hits you with his knife, it does a lot of damage so be careful. You may either use the Tonten to knock him out fast when he is about to use the knives, or the Talisman to freeze him then hit him from behind so he has to turn around to hit you with knives. And if you're wondering whether you can beat this boss using Conshi, well, YOU CAN. It's quite difficult however, and I'm not 100% sure if there are rewards somewhere, but it's quite fun watching him knock out that big guy =). +----------------------------------+ |Town 3 - Woo! A bit of difficulty!| +----------------------------------+ S I I S S T H O S O O B |_||_||_||_||_||_||_||_||_||_||_||_| Save the temple, get some scrolls, learn at the training hall, and then get the three orbs. Simple as that. LIGHTNING KYONSHI Difficulty: */*** --From here, the manual doesn't say the Japanese names, so you have to deal with the translation above. Sorry. --No, he doesn't throw balls of lightning at you. It's that the back- ground flashes every now and then. The boss itself is like any other tall Kyonshi, except he's much faster than the others, much stronger, and much more HP. +-------------------------------------+ |Town 4 - Hey! Be careful! WATCH OUT!!| +-------------------------------------+ H S S O S I O I T O I B |_||_||_||_||_||_||_||_||_||_||_||_| Don't bother learning the Wind Kick; it's pretty much useless. The other move, Mirage Thrust, is a must-have. You can easily kill tall bosses without even getting stratched. I know it's cheap, but an army of kyonshies vs. one phantom fighter isn't fair, too. TACKLING KYONSHI Difficulty: * / * --This is the part where I always got stuck when I was a kid. This boss attacks you at a speed in which you could rarely, if ever, react in time. If you don't hit him with anything in your arsenal, he WILL tackle you. Talisman does not work on him. The Tonten and the Sacred Sword does work on preventing him to attack, but does little to no damage. Try to put him into a corner then attack him whenever he goes floats back up so the chances of him tackling you will be minimized. +--------------------------------------------+ |Town 5 - Who shakes in terror? You, or them?| +--------------------------------------------+ I \\* S I S I T I O O O B |_||_||_||_||_||_||_||_||_||_||_||_| As you notice, there is no training hall. That means it is at the other town, so collect scrolls till you reach the maximum of 99. EARTHQUAKE KYONSHI Difficulty: * */*** --Why the \"earthquake\"? Because when it lands on the ground, it actually shakes it so much that Kenchi falls over. Also, when you're hit by the claws, you are sent to the other side of the room. The good thing? You could use the Mirage Thrust to post a talisman on him, or just plain punch his knees till his life meter runs out. +-------------------------------------------------------------+ |Town 6 - THIS, has gotta be, the weirdest flying kick ever...| +-------------------------------------------------------------+ H T O O O B |_||_||_||_||_||_| The moment you enter, enroll at the training hall and avail of the powerful Jump Kick. It's a shame you would learn it this late in the game. You may choose to save the temple, or if you feel really almighty, just leave it there and snatch the orbs one by one until you can enter the boss lair... AXE KYONSHI Difficulty: * */*** --He does not throw axes at you; it's his melee weapon. He just holds it out in front of him, swinging it once his near enough. Don't punch him; use your feet both for kicking and dancing around him. +-----------------------------------------------------------+ |Town 7 - What?! I thought I could use magic in this game...| +-----------------------------------------------------------+ O O S S S T H S S S O B |_||_||_||_||_||_||_||_||_||_||_||_| No items. Save the last temple, collect scrolls this one last time, then learn 4 Thrust at the last training hall. After three orbs, fight the last boss... FIREBALL KYONSHI Difficulty: */*** --Obviously, he throws fireballs. Very easy to dodge; just jump high then drop behind him. If you learned every skill, you could do it the easy way (crouch then punch) or the exciting way (stand in front of him and flaunt your amazing abilities). Just don't get burned by fireballs, 'cause it HURTS. +------------------------------+ |Town 8 - The stupid final boss| +------------------------------+ O O O FB |_||_||\\_|---|\\_\\_| The --- represents a bridge, and FB is the FINAL BOSS. Are you ready? There are no temples in here, so once you enter a building, you have to make it till the end. There's no turning back. I just hope you really mastered those moves... OBO Difficulty: */*** --First time I said she was easy, I forgot you have to fight an army of assorted Kyonshies to get to her :). --You're fighting, for the second time (the first was a ghost, remember?), a non-kyonshi. Besides teleportation and magic fireballs, she does nothing but stand there, waiting for you to hit her. Unless you are cursed with kyonshi-slow reflexes, she is very disappointing. Sure wish she was a lot more powerful. You may watch the credits roll, but you won't get a new game. Now it's all over. Kyonshies will never bug you again =). --- ## ---> CREDITS My father, for buying me a cartidge a looong time ago (back in 1989); FCI, for making the game; Nintendo, for a really good console that'll last for ages; CJayC, for hosting the guide on his site, GameFAQs; SatelliteGeibor@aol.com, for the Kyonshi names; and you, for playing the game. --- Contributor Recognition: http://www.gamefaqs.com/features/recognition/37592.html --- +------------------------------------------------------------------+ Phantom Fighter FAQ |>>>>>>>>>>>>>>>>>>Copyright 2003 Aaron Madrinan<<<<<<<<<<<<<<<<<<<| +------------------------------------------------------------------+ --- Phantom Fighter FAQ by The Deacon e-mail:mmacinni@cs.oberlin.edu --- Before we start, even though they are probably out of business, the game \"Phantom Fighter\" for the NES is trademark copyright etc. FCI and whoever else made it. Feel free to toss this FAQ to the winds, just don't be a lamer and put your name on it; if you have anything to add, e-mail me and I'll insert your info (as well as credit you with the addendum). ## Contents I.Why is this in FAQ format? I. Why is this in FAQ format? I know the term \"FAQ\" gets bandied about a lot, but since I'm something of a wordsmith I feel that we should nail down exactly what a \"FAQ\" is. You are reading a \"FAQ\", a compilation of \"Frequently Asked Questions\". Christ, if you have net access you should know by now what a FAQ is. Anyhow, it's also a FAQ because walkthroughs are really lame. The way I see it, any kind of aid on a game signals one's inability to rise to the challenge, however in the greater scope things I figure we can call getting level passwords the \"little white cheat\", reading a FAQ a medium or venial cheat, and working the keyboard/mouse with one hand while holding the walkthrough in the other and looking at the screen only to perform the walkthrough's dictated actions is a \"mortal\" cheat. And besides, walkthroughs are really only the kind of thing I could see a reasonable person using in a game with some kind of plot to it. I know a fellow who isn't very good at video games at all, and with as little patience as he has, when he gets a game he gets the walkthrough and goes through the game just to see the plot. However, Phantom Fighter (how's bout we shorten that to \"PF\") has _no_ plot. Really. You'll see. II. What's the plot? I just told you. There isn't one. (see above). However, if you want a basic rundown of the pseudo-plot, well then here you are: You are Kenchi, martial artist and aspiring Phantom Fighter. With your trusty sidekick you travel from town to town seeking fortune and fame as a wandering hero, driving out evil hopping vampires and rescuing innocent townspeople using only your wits, a few sacred items, and your superior kung-fu. That's about the size of it. III. What's the point of the game? Well, as with all games, it's to have fun...:). Besides that, there doesn't really seem to be much of a point...if you were to say that the \"point\" of the game is to be defined as what is required to beat a game, then I suppose there is a point. That definition makes the point of this game thus: You go from town to town (there are 8 in all), driving out vampires with your fists (and feet) of fury. Along the way, you pick up sacred items and ancient scrolls (more on them later) and improve your kung-fu until the final confrontation with the evil witch Obe. (\"obay\") IV. Anything I should know before I start the game? Since I didn't have the manual I'll take the time to explain the basic game mechanics for you. A: Kick. (or, if you prefer, \"Boot to the Head\") B: Punch. Start: Pause the game. Select: Does nothing as far as I know. V. What's the basic pattern of the game? Basically, you'll come to a town with your sidekick in tow. The mayor (I suppose) comes out and kowtows to you, weeping with gladness; for they called the thunder and now they got it. When you enter town, you'll be in a far-off side-scroll view with Kenchi and his sidekick walking around near buildings. When in front of a door, push A or B to see a brief message telling you what's going on inside. (I'll explain them later.) To enter a house, stop in front of the door and press Up. When you enter the house, your energy will be displayed on the left side of the screen, and any enemies in the room will have theirs displayed on the right. As you will soon see, a purple bar means you have full health, and as you take hits the little boxes turn pink until you get wiped out (and your sidekick speaks ill of the dead). In this screen, pressing up makes you jump, down makes you crouch. You cannot attack while crouching. (A BIG PISSER IMHO) Obviously, when you see a vampire come hopping up at you, kick and punch it until it falls over and bursts into flame, then continue to the right. Events vary, but after getting to the end of the house you will be presented with either scrolls or a sacred item, sometimes information (other times just blather). Some houses contain lots of enemies, make your way to the end, defeat a moderately tough enemy, and collect a jewel that instantly refills your health. You'll see it sit one of the empty stands in the upper right corner of the screen. Collect three of these things and you will have the magical power necessary to break a sealed portal. When you have three gems, you can enter the building that is always all the way at the far right end of town. Inside is a town boss, the source of the vampire scourge (for this area, anyhow). Teach him a lesson (\"Don't think. Feeeeeeelll!\") and you'll enjoy being toadied to by your sidekick. Then the happy mayor will give you a password and you're on your way to the next town. VI. What the hell ARE these things I'm fighting? Those, gamer, are hopping vampires -- China's version of the ubiquitous super- natural parasitic entity that has frightened many a superstitious person for hundreds of years. I believe the game's term for them -- \"Kyonshi\" -- is Mandarin Chinese for \"hopping vampire\". However, it could be a bastardized Chinese put into a game for mainly Japanese players. I think in Cantonese the term is \"Jiangshi\". I'll refer to them throughout this document as \"Kyonshi\", simply because that's what the game calls them. What makes these guys different from their Romanian \"cousins\"? Well, their origins are slightly different. Kyonshi are \"undead\", as in those who have died, but somehow have had their souls trapped in their body instead of escaping to the Nine Heavens (or Hells) after death. This could be for any number of reasons; bad chi flow through the house, suicide, lots of angst, etc. Anyhow, because they have died, their bodies aren't alive, they're just \"undead\". This means that they're not as limber as they once were, hence the hopping; it's the only way they can get around. What motivates them? Not much. Unlike the Western vampire, a Kyonshi has little or no conscious thought. Its only thought is to attack living things and eat their bodies. Since they don't consume their victims whole their victims become vampires as well. Kyonshi don't speak, don't respond intelligently to problems. This is a noticable advantage that the phantom fighter has over his unnatural opponents. A nimble phantom fighter should dance about to keep his assailant's long fingernails (so sharp they're almost claws) at bay. For those interested in Kyonshi I'll provide a bibliography at the end of this FAQ -- they're pretty cool. It also helps to know something about them for later (see below). VII. Do the towns follow a basic pattern? Yes. For the most part they are a bunch of houses, a temple, a \"boss shack\", and a martial arts studio. You can always recognize the temple and the studio, they're the same in every town. The temple is a large building with pillars in the front and lots of pillars and Buddha statues inside. The studio is a long, one-level wooden building. There will be three places where you can get gems as well, and the rest of the little shacks round out the town with places to kill Kyonshi and get ancient scrolls. This seems as good a time as any to explain what the little messages you get when you press A or B in front of a house mean. \"Kyonshies are here\": Ahem. \"Kyonshies are not here\":Ahem. Ahem. Okay, besides the obvious, these mean there is just a basic bunch of Kyonshies in the house (or the house is empty). Fight through all of these and you'll get some ancient scrolls. Empty houses seem to have no point; you walk all the way to the end and some cowering peasant thanks you or says something worthy of the term \"non sequitir.\" \"This is a temple.\": What the Christ. It's a temple. \"There's danger in the air\": This means that inside is either a sacred item or one of the three gems you need to bust them ghosts. \"What, nothing's happening?\":or something like that. Basically one of those houses with goodies or gems in them that you've cleared out already. \"Open the sealenter\": Obviously an error in the programming, this should read something like \"Open the seal to enter\". This is the \"boss shack\" that you need the gems to get inside. \"Enter with courage\":When you have the gems, you can open the seal to the \"boss shack\" and instruct the Kyonshi on the finer points of kung fu. (\"I hope you were paying attention.\") VIII. From the looks of things, I'm not a very good phantom fighter. I just got my butt kicked. Any way to improve on this sad situation? Fortunately, yes. At the beginning it may seem like you'll have a lame kick and punch for the entire game, but fear not. You can improve to such powerful kung-fu that you'll easily steamroll enemies. So, how do you go about improving your skills? Easy. First, you have to do a little struggling; you need to earn ancient scrolls. Just pick a small house and go in. When you get some scrolls go to the studio. When you enter, a fat guard accosts you, and asks you if you understand that you won't get kung-fu lessons for free (This is basically to spare you from the annoying un-skippable dialog that follows if you don't want to enter after all). When you say that you understand, he asks you a question, to test your knowledge. A. What the hell is this? Copy protection? I have no idea why this in the game. I can only guess that at the time this was made, it made sense, or really was to protect from people pirating the game. I don't have the manual, so I don't know. Don't sweat it too much, the questions are pretty easy, often it's obvious what the correct answer is, and if you fail you can always try again. B. Fuck that. Just give me the questions and answers. Ok, I guess quiz games aren't your thing. Here are some of the questions that the fat guy asks, and their answers: (These are from memory, more to come) \"Name an FCI video game.\" I think all of these are FCI games, pick \"WCW\" just to be sure. \"Why do Kyonshies only come out at night?\" They hate the sun, obviously. \"How do you capture a Kyonshie?\" Although you never see it in the game, you use an Urn. \"What is a Kyonshies least favorite food?\" I don't know where this hell this came from, by process of elimination it's ice cream. When you answer correctly the fat guy lets you pass and you can enter the Master's chambers, where they gather for the feast. And though they stab it with their steely knives, they just can't kill the beast. Sorry. If you got that reference you'll know the special sort of anguish I feel now at selling my copy of that album. Anyhow, the master will train you in kung fu in exchange for ancient scrolls. Why, I don't know, but my motto is do what works. So, cough up scrolls and train in the martial arts. The different moves you can select are listed with their scroll cost to the left of them. X. What are the different kung-fu moves, how much do they cost, and what do they do? Good question, grasshopper. There are three basic move categories: \"moves\" (\"mv\"):your movement \"thrust\" :punching moves \"kick\" :ahem, kicking moves. Here's a breakdown by category: MOVES: Wolf Move (\"wolf mv.\") Cost: 2 scrolls What it does: Makes you move faster. Useful. Tiger Move (\"tiger mv.\") Cost: 6 scrolls What it does: see above. Mirage Move (\"mirage mv.\") Cost: 6 scrolls What it does: see above. Dragon Move (\"dragon mv.\") Cost: 18 scrolls What it does: see above. Mirage Walk (\"mirage wk.\") Cost: x scrolls What it does: Allows you to walk while crouching. Mirage Thrust (\"mirage th.\") Cost: 50 scrolls What it does: Allows you to punch while crouching. A terrific move. High Jump Cost: x scrolls What it does: you jump higher. Absolutely necessary for airborne vampires. Windmill Jump (\"wind jump\") Cost: x scrolls What it does: you jump still higher, and you flip in the air as you do. Allows you full range with airborne enemies and looks really cool. THRUSTS: 2 Thrust Cost: 2 scrolls What it does: Gives you two lightning-fast punches instead of one. 3 Thrust Cost: x scrolls What it does: Basically same as above, except now it's a 1-2-3 punch. 4 Thrust Cost: 90 scrolls What it does: Now, you punch both high and low, twice. The most powerful move in the game, believe it or not. KICKS: 2 Kick Cost: 6 scrolls What it does: the 1-2 kick. One for both ears. :) Side kick Cost: 18 scrolls What it does: you lean to the side and use your hips to put a little torque action into your kick. Good damage, another move you'll use right up to the end. Windmill Kick (\"wind kick\") Cost: 50 scrolls What it does: When you jump straight up in the air and kick, you'll spin around in a flurry of feet. Really only useful for airborne enemies, and doesn't always come off clean. Jump Kick Cost: 80 scrolls What it does: Run at your enemy and kick and you'll leap at them with both feet forward. (\"Mind your Manners!!!\") Great as an opening move, as well as for enemies who like to jump a lot. XI. What are some good fighting tactics? Stick and move. If you stand in one spot you'll get creamed, as more often than not enemies will soak one or two kicks and swipe you with them claws. This is most important at the beginning of the game, when you can't do anything but punch and kick, and move around a little bit. The main method of attack Kyonshies use is to take little hops at you, then when they get close enough to a big Kyonshie leap into your face. What you want to try to do is to place yourself at a point where the Kyonshie will be in midleap when it comes into kick range. Then, when it leaps, kick it, and it'll fall over. It won't die, but knocking it down gives you breathing room. You'll take a lot of hits from Kyonshies just steadily leaping towards you. That's their other fighting tactic...they just keep coming, soaking up the kicks and punches until they can cut you. If you can kick them enough times, however, they'll fall over. Different kyonshies take more or less kicks/punches in a row to be knocked down, size is one factor. Later, when you learn more kung fu, it will be easier to defeat enemies, although the enemies will get progressively harder as you go from town to town. One good move that will help you out a lot is the \"Wind Jump\". This allows you to leap high into the air (doing a cool kung-fu flip) above the reach of the dreaded hoppping demons. When enemies come at you, kick them once or twice, and if they don't fall down hop over them just before they get to you. A little practice helps, but soon you will be able to drop down right on the other side of them and kick them in the back of the head while they are turning around. Kyonshies are very stupid. It helps a lot to remember this, as the next trick shows. See, Kyonshies (as you will know if you read the above section on them) are in the throes of rigor mortis, their bodies in an arrested state of decay. So they can only see straight forward, right? So if you crouch down, they will turn back and forth, their tiny minds trying to figure out how you pulled your disappearing act. This will not be very useful in the beginning, as you can't walk while crouching. DON'T try this just before a Kyonshie gets to you -- their turning back and forth motion will still be able to hit you, as you are not completely under their arm level (unless you are fighting a really tall kyonshie, see below). This move becomes more useful when you learn the Mirage Walk, as you'll be able to shuffle about under their vision, just don't attempt to get too close to them as they can often still hit you. Later, when you learn Mirage Thrust, this becomes a winning tactic, allowing you to stand under tall enemies and bosses and punch them in their unfeeling vampire gonads mercilessly. (WARNING: This is kind of a cheapo move, as you really can kill tall kyonshi bosses with this tactic. It certainly doesn't _look_ very exciting.) XII. Alright. I'm in town. What do I do first? Wander through town until you find the temple, then try to fight your way through it, shouldn't be to difficult. In the first town, it's one of the first buildings, later on it will be farther into town, or sometimes nonexistent. Anyhow, it's important to start here, so that you can fill up your health, you'll need to often and there's only one other way to do it in the game, and that involves more fighting. Once you've done that, go to the kung-fu studio and see how many scrolls you'll need to learn the various kung-fu moves. The dialog trees can be annoying, but get used to them because they are all through the game. :P Anyhow, leave the studio with the moves you want to learn in mind, and find a house with Kyonshies in it. Then enter and work your way to get the scrolls. Surprisingly (or perhaps not, depending on how well versed you are in \"game logic\"), you can enter the same house again and again, continuously obtaining scrolls. You can get up to 99, but don't bother trying that here unless you are sick in bed with chicken pox or something -- you'll have better oppportunities later on when you recieve lots more scrolls for rescuing townspeople. Learn all the moves you can, when the master tells you you've tapped him out, you can move on. A. Just to make it easier on myself, what moves should I learn first? Since your kung-fu is puny, learn attack moves first. Specifically, get 2 Kick and 2 Thrust. Throughout the game, when you have an opportunity to learn an attack move, learn it, and learn the other moves later. While the \"Move\" moves may seem pointless, they are actually very useful, since being able to move quickly will allow you to dance in an out of a kyonshie's range without getting creamed every time you try to hit them. It's also highly important to learn the jump moves, since they allow you to attack airborne enemies, of which there are 3 in the game, all of them important. While I suppose you could beat the game without all of the moves, I wouldn't recommend it, simply because it wouldn't be as fun, as learning new moves is one of the high points of the game. After you've learned all the moves you care to, head for the houses with the gems in them, collect the gems, and head for the boss shack. Don't forget to write down your password! XIII. Hey, these Kyonshies aren't all alike! Quite right. There are several different types of Kyonshies. Basically, there are four basic types of Kyonshi, differentiated by body type: the little one, the medium one, the tall one, and the big fat one. Within these different morphical categories there's lots of room for variation in the different colors they wear. You'll no doubt become quite familiar with the types as you play the game, and certain ones will become infamous. (\"ARgh! Not more green ones!\") XIV. Okay, so there's different types. What are they like? Without further ado: little: This little bastard is fast and hard to hit. He'll be a toughie when you're duking it out in the first town and don't have awesome kung-fu. His basic weakness is that he can't jump that far, and he's really short, making him easy to hop over. Fortunately he's still just tall enough to kick in the face. medium: Your basic vanilla Kyonshi. Some types jump a lot, some are fast, etc. tall: The tall guy moves ponderously slow, and thanks to his height is hard to jump over. He also jumps very high while attacking as well. Naturally enough this is a big mistake on his part; he jumps so high that you can crouch and let him go over you, then \"boot to the head\" while he's turning around. Later on when you learn the Mirage Thrust the tall guy will be at your mercy as your firsts pound mercilessly into his undead loins, and his turning back and forth schtick won't be able to nail you. fat: This guy is tougher than he looks. He's very strong, such that even though he moves slowly, he packs a real wallop. Also, he doesn't jump that high, but is still tall enough that jumping over him is a bit of a sticky wicket. Your big advantage over him is speed. While he can soak up hits and really wummox you when he gets a hit in, if you play it right you can kick once or twice then get the hell out of the way when the Kyonshi express comes through. XV. I won some kind of item, what does it do? Aha, one of the coveted mystical items that you can use in the game. Don't confuse these with scrolls -- ancient scrolls sound cool but don't do jack except buy you kung-fu. However, the items are quite nice. First, here's how to use an item once you've got it. Whenever you obtain an item, you will see it appear at the top of the screen, in a long box to the left of the stands that hold the three gems you'll need to get out of town. However, make no mistake, you are NOT carrying that item. Your faithful sidekick (who has no name that I could tell) is carrying them. Thus, whenever you enter a building, and you want to use an item, turn back as if to leave. Your boy will pop his head in and ask if you want to split, use an item, or ignore him and get back to butt-kicking. Well, ask for an item, and he'll give you a list: strangely enough, he lists all the items you can get, even if you don't have them yet. Pick one, and it will be outlined by a red square at the top of the screen. This means you are now carrying the item, and will use it instead of punching when you push A. Now, here's all the items, and what they do. Talisman: Freezes enemies in their tracks sometimes. Tonten: Knocks baddies onto their cans with a flash of light. Sword: Knocks guys down, you can also slice 'em with it. Bell: Controls the demon-boy. A note of warning: your items will also take hits for you if you get hit. However, this will also break the item, making it useless. Hey, it's an antique. Keep this in mind when fighting with an item. It's a good idea to grab them, they're nice for keeping enemies at bay, and the Tonten is pretty damn invaluable for beating the game. XVI. I bought one of those Game Genies. Any codes for this game? CODE KEY IN . . . EFFECT . . . 1 VTVKEGSA + KAVKOGNA Start with Sword 2 VTVKEGSA + SAVKOGNA Start with Bell 3 VTVKEGSA + UAVKOGNA Start with Tonten 4 VTVKEGSA + XAVKOGNA Start with Talisman 5 LASKNGAA + VAVKOGNA Start with 3 Scrolls 6 TASKNGAA + VAVKOGNA Start with 6 Scrolls 7 SXSZLUSE Infinite energy 8 OVSZPLSV + PESZZLAA Take less damage when attacked (author's note: This is taken verbatim from Galoob's master list of all Game Genie codes for NES games that they ever published. There may, of course, be some home- grown codes floating about. So all due credit goes to Galoob for the above blurb.) XVII. This isn't the most complete FAQ I've ever read, you know... Sorry. This is mostly from memory. If I get e-mails requesting updates I'll saunter back into the game with my rightfully earned cheat codes and research it fully for those who desire it. And besides, if you think this FAQ is incomplete, check out some of the other (admittedly) sparse fare at TSR's NES FAQ page. For right now I'll call an end to this FAQ, just because I figure this is enough to be written about any NES game that wasn't made by Square. :) I'll send my regards first and foremost to TSR, who's NES page is truly one of the best place for NES info on the 'net, not to mention skilled use of graphics and delightful content like the NES oddity page. Also, here's to the place that posted the ROM I used for this FAQ, Big Daddy's International House of ROMs, THE place to get ROMs on the 'net. In my experience, if he doesn't have it here, it doesn't exist yet (at least for NES, anyhow). I'll also send out greets to John Turk, who's NES Underground Library is probably the most ambitious of all the NES pages I've seen. And finally I'll send shots out to all the other NES pages, all those who've stayed cool despite NESticle's appearance, and to all the fellows on IRC channels like #emuroms, #emulator, and #emu who are out there circulating coolness in less than 200k. :) Also a shot out to my friends on #1980Warez, who have nothing to do with NES ROMs, but are cool anyway. Jibes go to all those who feel the need to charge excessive amounts of money for their emulators. Say what you will about effort and reward, guys, but in the end you're creating something used to do something that's, shall we say, a trifle off-color in the eyes of the law. Jibes also go to those who run IRC channels so huge that no chatting goes on in them, and channels that have nothing but DCC bots in them. Thanks for helping turn IRC into a wasteland, guys. So much have I written for gamers, now give me a drink! The Deacon Restore Page Training Hall Quiz Questions and Answers +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^+ < ////// // // /// //// // ////// ///// ////////// > < // // // // // // // // // // // // // // // > < ///// ////// /////// // //// // // // // // // > < // // // // // // /// // // // // // // > < // // // // // // // // ///// // // // > < > < ////// // ///// // // ////// ////// ////// > < // // // // // // // // // > < //// // // /// ////// // //// //// > < // // // // // // // // // // > < // // ///// // // // ////// // // > +------------------------------------------------------------------+ * * * * * * Training Hall Quiz Questions and Answers * * * * * * A guide by PinKirby (Pinku_Kirby@yahoo.com) +-----------------+ ----Version 1.0---- +-----------------+ June 9th, 2004 -Began this guide -------------------- Contents: 1. Introduction 2. The Training Hall 3. The Questions and Answers 4. Possible Messages 5. Thanks 6. Legal Info --------------------- * * * * * * 1. Introduction * * * * * * Phantom Fighter is a game that is not very well-known, but is still, in my opinion, a great game. I've decided to make a guide on the questions the guard of the Training Hall asks you. Most questions have one correct answer, but some questions are asked in which ALL of the multiple choices are correct answers. Here's how to use this guide, using a sample question NOT asked in the game: ***QUESTION: What game does this guide cover? 1. Super Mario Brothers (X) 2. Phantom Fighter (O) 3. This game has no name(X) (X) - Means an incorrect answer, in which the guard throws you out. (O) - Means a correct answer, in which the guard lets you pass. --------------------------------------------------------------- * * * * * * 2. The Training Hall * * * * * * After defeating the monsters, the sharp-nailed Chinese phantoms which are known as \"Kyonshies\", in houses and buildings, you can get one or more scrolls. You use these scrolls to learn skills in the Training Hall, but you can't see the master for free, as his guard explains to you. To prove you are worthy of the master's time and attention, you must correctly answer a question with one of three multiple choices. While many of the guard's questions pertain to your foes the Kyonshies, some of the questions involve trivia. If you get a question wrong, DON'T FRET! You can re-enter and try again with another question. The guard tells you, Kenchi: \"You can't see the master for free. Do you understand?\" If you answer no, he'll laugh at your cowardice and kick you out. Answer yes, and it's quiz time! --------------------------------------------------------------- * * * * * * 3. The Questions and Answers * * * * * * QUESTION 1: How many stars are there in the American flag? 1. 50 (O) 2. 28 (X) 3. 5 (X) --------------------- QUESTION 2: What country do samurais come from? 1. India (X) 2. Japan (O) 3. Spain (X) --------------------- QUESTION 3: What is the best method to make sure Kyonshies never revive again? 1. Big Stakes (X) 2. Seal in Rock (X) 3. Fry in Oil (O) --------------------- QUESTION 4: What is the teaching taught by Confucius called? 1. Psychiatry (X) 2. Confucianism (O) 3. Physiognomy (X) --------------------- QUESTION 5: Who built the Great Wall of China? 1. The Emperor (O) 2. Plasterer (O) 3. Farmer (O) --------------------- QUESTION 6: What is Kyonshies' most powerful weapon? 1. Sharp Claws (O) 2. Sharp Eyes (X) 3. Swift Moves (X) --------------------- QUESTION 7: What is the horrible skill called when your head is banged against your opponent's head? 1. Head Strike (O) 2. Head batting (O) 3. Head to Head (O) --------------------- QUESTION 8: What's the name of George Bush's dog? (This is George Bush Sr.!) 1. Millie (O) 2. Martha (X) 3. Pooch (X) --------------------- QUESTION 9: Who created Ultima? 1. Lord British (O) 2. Prince Chuck (X) 3. Mr. Exodus (X) --------------------- QUESTION 10: What is the best thing to use to capture Kyonshies? 1. Coffin (X) 2. Urn (O) 3. Jewelry Box (X) --------------------- QUESTION 11: What is the famous Chinese Newspaper? 1. School Paper (X) 2. Wall Poster (O) 3. Daily China (X) --------------------- QUESTION 11: What is the food Kyonshies hate? 1. Liver (X) 2. Ice Cream (O) 3. Frog Eyes (X) --------------------- QUESTION 12: Something is used to beat Kyonshies. What is it? 1. Bird's Blood (O) 2. Lizard Scale (X) 3. Vulture Nail (X) --------------------- QUESTION 13: Why do Kyonshies only come out at night? 1. Play late (X) 2. Shy (X) 3. Hate the Sun (O) --------------------- QUESTION 14: Which emperor is often called the Last Emperor? 1. Sagi (X) 2. Higi (X) 3. Fugi (O) --------------------- QUESTION 15: What kind of place do Kyoshies usually live in? 1. Wet place (O) 2. New Jersey (O) 3. Beverly Hill (O) --------------------- QUESTION 16: What is the skill called when you hold your opponent's arms from the back and throw him backward? 1. Dragon Suplex (O) 2. Front Suplex (X) 3. Side Suplex (X) --------------------- QUESTION 17: Name an FCI Video Game. 1. Hydlide (O) 2. WCW (O) 3. Ultima (O) --------------------- QUESTION 18: What is a Chinese martial art usually called? 1. Kung Fu (O) 2. Martial Art (X) 3. Phantom Kick (X) --------------------- QUESTION 19: What is another word for one? 1. Uno (O) 2. Ichi (O) 3. Un (O) --------------------- --------------------------------------------------------------- * * * * * * 4. Possible Messages * * * * * * ----- CORRECT MESSAGES: - That's good. The master is waiting for you. Go and see him. If you answer \"New Jersey\" or \"Beverly Hill\" to the question about where Kyonshies usually live in: - That's right. Now as you know, that is the hangout of Kyonshi. Go and enter. (I'm assuming the guard pities you so much that he just accepts it as a correct answer. :P ) ----- INCORRECT MESSAGES: - Why are you trying to pretend you don't know the answer? - You, Kenchi, not know a thing like this? Worth a laugh! Ha ha ha! Now, get out! Get out! - That was a pity. Train yourself and come again. - How little you know I'm stunned. Think it over and come back again. - Kenchi, you are really an idiot! Go home, try to learn more, and come back when you are wiser. If you answer \"5\" as to how many stars are on the US flag: - Dunce! --------------------------------------------------------------- * * * * * * 5. Thanks * * * * * * - To my sister, Mom and Grandpa - To FCI and Pony Canyon for making this game - To YOU for taking the time to read this guide! * * * * * * 6. Legal Stuff * * * * * * This guide is copyrighted (c) by PinKirby. You may copy SOME of this guide, but you MUST give me, PinKirby, credit. If you want to put this guide up on your website, email me and I will see about it. If I find that this guide is on a website NOT listed below, then I will have it removed. SITES THAT MAY HAVE THIS GUIDE UP: www.GameFaqs.com Phatom Fighter is (c) 1989 by FCI and Pony Canyon. The Nintendo Entertainment System and Nintendo of America are registered trademarks of Nintendo. All rights reserved. This Phantom Fighter Training Hall Quiz Guide is (c) by PinKirby Restore Page","title":"nes"},{"location":"nes/#nes-faq","text":"","title":"nes faq"},{"location":"nes/#phantom-fighter","text":"==================================================================== Phantom Fighter Complete FAQ/Walkthrough September 05, 2003 Version 2.00 author: Aaron Madrinan (snoocete) e-mail: snoocete@yahoo.com subject: \"RE: Phantom Fighter FAQ/Walkthrough\" ==================================================================== --- ## ---> INTRODUCTION The gaming world is changing very rapidly. Games are released on breakneck speed that we need more than a lifetime to play even just the good ones. The result? The other not-so-famous games are forgotten and put aside by future gamers. In other words, they become underrated. Such is what happened to a game called Phantom Fighter. I've played it since I was a kid. Though not as extremely fun as games like Kirby or Pokemon, it very much dwells on the concept in which Contra flourished: you can never let the game beat you. Supposedly, it would be as famous as that. But, heck there wasn't even a walkthrough on GameFAQs, only some codes that lets you skip to the last town and a very bad review. Sigh. You want my opinion? Here is what I can say: don't let the big names fool you. A lot of gems are hidden in the dump of really bad games, I admit, but they only need a little spotlight to shine. Here is my share of that spotlight =). --- ## ---> DISCLAIMER You are free to save this guide onto your hard drive and/or print for your personal viewing pleasure. You may distibute it to other people as long as you don't claim it as yours, okay? Since this game doesn't have that much coverage, you may post this at your site in its full, unedited version even without my permission as long as its credits point to me. Deal! --- ## ---> CONTENTS 1. Updates --- 2. FAQ --- 3. Walkthrough --- 4. Credits --- This is FAQ #2, hope I got a bit better. --- ## ---> UPDATES -2.00- -21Kb- Look above (^\\_^). It's a whole number change! A kind soul mailed to me the names of the Kyonshies. Yay! Added peace and order (where there was chaos). Formatting change. E-mail change. Lastly, look at the disclaimer. -1.00- -19Kb- This FAQ is complete, however the Japanese names of the Kyonshies are missing. Not that anybody would care, but it still important. If you have an instruction manual (because I kinda lost it), kindly e-mail me and let me know, ok? --- ## ---> FAQ --Who are you? Just a humble freelance (a bit newbie-ish, but wants to learn a lot) FAQ writer who writes for his favorite games whenever he has the time. --What are these \"Kyonshi\" that I'm fighting? If you have ever watched the anime series \"Shaman King\", or knew a bit about Chinese folkore, you have the idea. These are what we call the undead; they are supposed to be immobile, rotting bodies. They still are, but they are being controlled by some mysterious power. You know about rigor mortis? It also applies to these phantoms. That explains why they could only move their feet and knees to hop and turn around, and their elbows to puncture your face with their dirty, sharp nails. --How do I fight these enemies? You have to move around. A lot. Whenever you stay on one spot, the Kyonshi will follow you with small hops, then when it gets in range, it suddenly does a big dive toward you. Unless you hit the Kyonshi in a critical point or you kill it in transit, you cannot stop that from homing to you. So you got to move. That's your ace against these creatures. They have lockjaw and every known bone and muscle disease, while you could just move out of their way. For example, the moment they land on the ground, give them a kick or two on their sleeves then they will be knocked down. When they wake up and hops around again, repeat. Soon they will burst in flames, and the door that lets you continue through the building opens. --What does that man who follows me around do? That's your assistant. Whenever you are inside a building, if you return to the exit, you talk to him. He asks if you want to have an item, stay at the building, or just leave the place. --And those items are...? There are a total of four (4) items in the game. You can find them from the households you rescue from kyonshies. They replace the use of the punch attack, but it's worth it. 1. Tonten - the kyonshi you show this with is knocked back to the ground because a) there is holy white light coming from it, or b) he saw his face, squirming with maggots (hey, thanks mate for the mirror, 'bout time I fix my hair... what the he... _faints_). DON'T use this when you're about to get hit; it also acts as your armor, and because it's a mirror and was hit by diamond hard claws, you know what happens. 2. Sacred Sword - even a swordless Link (of Legend of Zelda) would not use this nicely-named weapon (he might even throw it back at you so don't even try). It does little damage, and it also gets broken easily as if it is made by glass. The only edge it has is that it has a rather long range for a physical attack, you could corner a kyonshi with it, but you would like to beat them up yourself, wouldn't you? 3. Talisman - freezes the enemy in place until it's effect is gone or they got hit. You have to make contact with the kyonshi you want to victimize it with, so there is a chance you get hit and the item breaks. Useful on the tall kyonshies, you could sneak under them, paste the talisman on their belly, then kick them in the butt. 4. Bell - If you use this in the middle of a heated battle, it does nothing except probably makes a little chime that is pleasant to hear for you but instead boils the blood of the kyonshi, making him more aggressive. Actually, use it to control the kyonshi kid. --What are the scrolls for? Whenever you cleanse a building from kyonshies, you get a varying amount of scrolls. You use them at the Training Hall to learn new abilities. --The guard won't let me in because I can't answer his questions... got a list there that could help me? Please? Of course I do. I intend to make this the complete-est FAQ you'll ever see (for PF, of course). Here they are (if you noticed, the stupid comments are gone. just learned not to learn from the likes of ArchNacho and Tortilla Godzilla): -Format- Q - Question A - Answer Q: Why do Kyonshies come out only at night? A: Hate the sun. Q: What country do Samurais come from? A: Japan Q: What is another word for one? A: Any answer will do. Q: Who built the Great Wall of China? A: Any answer will do. Q: What is the food Kyonshies hate? A: Ice Cream Q: Who created Ultima? A: Lord British SC: Who is he? Q: What is Kyonshi's most powerful weapon? A: Sharp claws Q: What is the skill when you hold an opponent's arms from the back and throw him backward? A: Dragon Suplex Q: What kind of place do Kyonshies usually live in? A: Any answer will do. Q: What is a horrible skill when you head-banged against an opponent's head? A: Any answer will do. Q: Name an FCI video game. A: Any answer will do. Q: What is the famous Chinese newspaper? A: Wall poster Q: How many stars are there in the American flag? A: 50 Q: What is the best thing used to capture Kyonshies? A: Urn Q: What is a Chinese martial art usually called? A: Kung fu Q: What is the best method to make sure Kyonshies never revive again? A: Fry in oil Q: What is the teaching taught by Confucius called? A: Confucianism Q: What's the name of George Bush's dog? A: Millie Q: Who is the emperor called the Last Emperor? A: Fugi --Can I kill the guard? If you are not a super nerd, who knows NES rom-editing to change the layer which the guard resides to your characters layer and set its attributes to \"hittable\", has an HP count, and many other things, definitely no. I know it's a big disappointment. --What are these new abilities you will learn from the Training Hall? New techniques in martial arts. When you started the game, you have a weak punch and a weak kick. Learn these and you will become stronger and better to fight kyonshies. Town 1 (2scr) =2 Thrust - punch and you will hit the enemy twice =2 Kick - same with 2 Thrust but you kick instead =High Jump - self-explanatory =Wolf Move - speed up Town 2 (6scr) =3 Thrust - three fists-of-fury in one swift delivery =Turn Kick - most versatile move in the game =Wind Jump - jump and hold left/right to do a spin jump =Tiger Move - more speed up =Mirage Move - combine with Tiger Mv. for better effect Town 3 (18scr) =Mirage Walk - move while crouching =Dragon Move - whoosh! last speed upgrade Town 4 (50scr) =Wind Kick - jump then kick furiously in mid-air to form a spin kick =Mirage Thrust - punch while crouching Town 5 = Town 6 (80scr) =Jump Kick - hold left/right then kick to see this fantastic move Town 7 (90scr) =4 Thrust - most powerful move in the game Town 8 = --What are the names of these Kyonshies? What I have here are unofficial names which I made up by their looks, color, and strength: Pink - weak household form of kyonshi. Green - stronger and faster than the pink Dwarf - a real pain, their size and speed make them hard to hit. Tall - vulnerable to punch attacks, mirage thrust Fat - looks tough but slow, easy target Short-armed - has a high resistance to hits, has high attack power Their original Japanese names (courtesy of SatelliteGeibor@aol.com): Pink Kyonshi: Sosekushi Green Kyonshi: Zanshi Fat Kyonshi: Kimenshi Dwarf Kyonshi: WeeKyonshi (literally) Tall Kyonshi: Ryukyoshi Graveyard Ghost: Shanshi --Know any GameGenie codes? Nope. Go to the Codes and Secrets page of this game and you might find something to help, you cheater! --- ## ---> WALKTHROUGH In case you don't have a manual, here is the juicy bit of the story: You are Kenchi, China's savior from the Kyonshi, the eastern counter- part of the vampire. Someone had let them loose on seven towns, with the eighth one being totally controlled by that being. Of course, you as the hero walks in the scene and puts a stop to the onslaught. However, it won't be easy: the Kyonshi are undead, has little sense of pain and their conscience all gone. Which makes them more fun to battle, isn't it? +---------------------------------+ |Town 1 - A taste of Kyonshi power| +---------------------------------+ Enter the temple. This holy building looks always the same in every town you come to; it also functions the same thing; as a rest point for your character. That's why you should also invade this first. Oh, the controls: left/right: moves you in the direction you press. up: jump (not so high right now, so don't use it much) down: crouch/duck B button: punch A button: kick Select: when outside, changes message display speed Start: pauses the game Kill the introductory kyonshi inside, then the temple could now be used as an inn. The ASCII art below shows the lay-out of the town: T S O S S O H S O I I B |_||_||_||_||_||_||_||_||_||_||_||_| Legend: T-Temple H-Training Hall I-Item inside S-Scroll inside O-Orb inside B-Boss inside Enter houses and fight kyonshies to have scrolls and some items. Pink kyonshies are weaker than green ones, but later in the game, most of the time you will fight green ones so you have to practice on them, too. Heal at the temple if you are hurt. Every time you get enough scrolls, head to the Training Hall and learn a new ability, in the order 2 Kick, Wolf Move, 2 Thrust, then High Jump. Afterwards, enter the Orb buildings, defeat the more difficult kyonshies there and get the Orb they leave. After you have three orbs, enter the boss cave and fight... AFTERIMAGE KYONSHI (Japanese name: Genyoshi) Difficulty: */ * --I just made up the name, sorry. This boss is small but jumps ultra high. Also it takes up many hits to bring down so don't you dare stand in its path to kick/punch. You could a) wait until it reaches the high point in its jump, then as it falls down, attack it, or b) my favorite way. Use the talisman to freeze it then kick it from behind. Rinse and repeat. Your reward? Being congratulated by your assistant and access to the next town =). Don't forget to copy the password. +-------------------------+ |Town 2 - The baby Kyonshi| +-------------------------+ I H S S O O T I O \\* I B |_||_||_||_||_||_||_||_||_||_||_||_| The \\* contains Conshi, the baby Kyonshi. You can have him in your party by letting your assistant be snatched by the light in the graveyard, defeating her (she's very strong, but not too hard), then afterwards you will get the bell. You may want to try it to know how it feels to be a Kyonshi; otherwise do the same modus operandi in Town 1 (learn the Turn Kick as soon as you can), and at the end is another cave... KNIFE KYONSHI (Japanese name: Raunshi) Difficulty: /*** --As soon as you enter, stop for the boss is throwing knives at you. After enough knives, he pauses, that's your signal to get near and whack him. If he hits you with his knife, it does a lot of damage so be careful. You may either use the Tonten to knock him out fast when he is about to use the knives, or the Talisman to freeze him then hit him from behind so he has to turn around to hit you with knives. And if you're wondering whether you can beat this boss using Conshi, well, YOU CAN. It's quite difficult however, and I'm not 100% sure if there are rewards somewhere, but it's quite fun watching him knock out that big guy =). +----------------------------------+ |Town 3 - Woo! A bit of difficulty!| +----------------------------------+ S I I S S T H O S O O B |_||_||_||_||_||_||_||_||_||_||_||_| Save the temple, get some scrolls, learn at the training hall, and then get the three orbs. Simple as that. LIGHTNING KYONSHI Difficulty: */*** --From here, the manual doesn't say the Japanese names, so you have to deal with the translation above. Sorry. --No, he doesn't throw balls of lightning at you. It's that the back- ground flashes every now and then. The boss itself is like any other tall Kyonshi, except he's much faster than the others, much stronger, and much more HP. +-------------------------------------+ |Town 4 - Hey! Be careful! WATCH OUT!!| +-------------------------------------+ H S S O S I O I T O I B |_||_||_||_||_||_||_||_||_||_||_||_| Don't bother learning the Wind Kick; it's pretty much useless. The other move, Mirage Thrust, is a must-have. You can easily kill tall bosses without even getting stratched. I know it's cheap, but an army of kyonshies vs. one phantom fighter isn't fair, too. TACKLING KYONSHI Difficulty: * / * --This is the part where I always got stuck when I was a kid. This boss attacks you at a speed in which you could rarely, if ever, react in time. If you don't hit him with anything in your arsenal, he WILL tackle you. Talisman does not work on him. The Tonten and the Sacred Sword does work on preventing him to attack, but does little to no damage. Try to put him into a corner then attack him whenever he goes floats back up so the chances of him tackling you will be minimized. +--------------------------------------------+ |Town 5 - Who shakes in terror? You, or them?| +--------------------------------------------+ I \\* S I S I T I O O O B |_||_||_||_||_||_||_||_||_||_||_||_| As you notice, there is no training hall. That means it is at the other town, so collect scrolls till you reach the maximum of 99. EARTHQUAKE KYONSHI Difficulty: * */*** --Why the \"earthquake\"? Because when it lands on the ground, it actually shakes it so much that Kenchi falls over. Also, when you're hit by the claws, you are sent to the other side of the room. The good thing? You could use the Mirage Thrust to post a talisman on him, or just plain punch his knees till his life meter runs out. +-------------------------------------------------------------+ |Town 6 - THIS, has gotta be, the weirdest flying kick ever...| +-------------------------------------------------------------+ H T O O O B |_||_||_||_||_||_| The moment you enter, enroll at the training hall and avail of the powerful Jump Kick. It's a shame you would learn it this late in the game. You may choose to save the temple, or if you feel really almighty, just leave it there and snatch the orbs one by one until you can enter the boss lair... AXE KYONSHI Difficulty: * */*** --He does not throw axes at you; it's his melee weapon. He just holds it out in front of him, swinging it once his near enough. Don't punch him; use your feet both for kicking and dancing around him. +-----------------------------------------------------------+ |Town 7 - What?! I thought I could use magic in this game...| +-----------------------------------------------------------+ O O S S S T H S S S O B |_||_||_||_||_||_||_||_||_||_||_||_| No items. Save the last temple, collect scrolls this one last time, then learn 4 Thrust at the last training hall. After three orbs, fight the last boss... FIREBALL KYONSHI Difficulty: */*** --Obviously, he throws fireballs. Very easy to dodge; just jump high then drop behind him. If you learned every skill, you could do it the easy way (crouch then punch) or the exciting way (stand in front of him and flaunt your amazing abilities). Just don't get burned by fireballs, 'cause it HURTS. +------------------------------+ |Town 8 - The stupid final boss| +------------------------------+ O O O FB |_||_||\\_|---|\\_\\_| The --- represents a bridge, and FB is the FINAL BOSS. Are you ready? There are no temples in here, so once you enter a building, you have to make it till the end. There's no turning back. I just hope you really mastered those moves... OBO Difficulty: */*** --First time I said she was easy, I forgot you have to fight an army of assorted Kyonshies to get to her :). --You're fighting, for the second time (the first was a ghost, remember?), a non-kyonshi. Besides teleportation and magic fireballs, she does nothing but stand there, waiting for you to hit her. Unless you are cursed with kyonshi-slow reflexes, she is very disappointing. Sure wish she was a lot more powerful. You may watch the credits roll, but you won't get a new game. Now it's all over. Kyonshies will never bug you again =). --- ## ---> CREDITS My father, for buying me a cartidge a looong time ago (back in 1989); FCI, for making the game; Nintendo, for a really good console that'll last for ages; CJayC, for hosting the guide on his site, GameFAQs; SatelliteGeibor@aol.com, for the Kyonshi names; and you, for playing the game. --- Contributor Recognition: http://www.gamefaqs.com/features/recognition/37592.html --- +------------------------------------------------------------------+","title":"Phantom Fighter"},{"location":"nes/#phantom-fighter-faq","text":"|>>>>>>>>>>>>>>>>>>Copyright 2003 Aaron Madrinan<<<<<<<<<<<<<<<<<<<| +------------------------------------------------------------------+ --- Phantom Fighter FAQ by The Deacon e-mail:mmacinni@cs.oberlin.edu --- Before we start, even though they are probably out of business, the game \"Phantom Fighter\" for the NES is trademark copyright etc. FCI and whoever else made it. Feel free to toss this FAQ to the winds, just don't be a lamer and put your name on it; if you have anything to add, e-mail me and I'll insert your info (as well as credit you with the addendum). ## Contents I.Why is this in FAQ format? I. Why is this in FAQ format? I know the term \"FAQ\" gets bandied about a lot, but since I'm something of a wordsmith I feel that we should nail down exactly what a \"FAQ\" is. You are reading a \"FAQ\", a compilation of \"Frequently Asked Questions\". Christ, if you have net access you should know by now what a FAQ is. Anyhow, it's also a FAQ because walkthroughs are really lame. The way I see it, any kind of aid on a game signals one's inability to rise to the challenge, however in the greater scope things I figure we can call getting level passwords the \"little white cheat\", reading a FAQ a medium or venial cheat, and working the keyboard/mouse with one hand while holding the walkthrough in the other and looking at the screen only to perform the walkthrough's dictated actions is a \"mortal\" cheat. And besides, walkthroughs are really only the kind of thing I could see a reasonable person using in a game with some kind of plot to it. I know a fellow who isn't very good at video games at all, and with as little patience as he has, when he gets a game he gets the walkthrough and goes through the game just to see the plot. However, Phantom Fighter (how's bout we shorten that to \"PF\") has _no_ plot. Really. You'll see. II. What's the plot? I just told you. There isn't one. (see above). However, if you want a basic rundown of the pseudo-plot, well then here you are: You are Kenchi, martial artist and aspiring Phantom Fighter. With your trusty sidekick you travel from town to town seeking fortune and fame as a wandering hero, driving out evil hopping vampires and rescuing innocent townspeople using only your wits, a few sacred items, and your superior kung-fu. That's about the size of it. III. What's the point of the game? Well, as with all games, it's to have fun...:). Besides that, there doesn't really seem to be much of a point...if you were to say that the \"point\" of the game is to be defined as what is required to beat a game, then I suppose there is a point. That definition makes the point of this game thus: You go from town to town (there are 8 in all), driving out vampires with your fists (and feet) of fury. Along the way, you pick up sacred items and ancient scrolls (more on them later) and improve your kung-fu until the final confrontation with the evil witch Obe. (\"obay\") IV. Anything I should know before I start the game? Since I didn't have the manual I'll take the time to explain the basic game mechanics for you. A: Kick. (or, if you prefer, \"Boot to the Head\") B: Punch. Start: Pause the game. Select: Does nothing as far as I know. V. What's the basic pattern of the game? Basically, you'll come to a town with your sidekick in tow. The mayor (I suppose) comes out and kowtows to you, weeping with gladness; for they called the thunder and now they got it. When you enter town, you'll be in a far-off side-scroll view with Kenchi and his sidekick walking around near buildings. When in front of a door, push A or B to see a brief message telling you what's going on inside. (I'll explain them later.) To enter a house, stop in front of the door and press Up. When you enter the house, your energy will be displayed on the left side of the screen, and any enemies in the room will have theirs displayed on the right. As you will soon see, a purple bar means you have full health, and as you take hits the little boxes turn pink until you get wiped out (and your sidekick speaks ill of the dead). In this screen, pressing up makes you jump, down makes you crouch. You cannot attack while crouching. (A BIG PISSER IMHO) Obviously, when you see a vampire come hopping up at you, kick and punch it until it falls over and bursts into flame, then continue to the right. Events vary, but after getting to the end of the house you will be presented with either scrolls or a sacred item, sometimes information (other times just blather). Some houses contain lots of enemies, make your way to the end, defeat a moderately tough enemy, and collect a jewel that instantly refills your health. You'll see it sit one of the empty stands in the upper right corner of the screen. Collect three of these things and you will have the magical power necessary to break a sealed portal. When you have three gems, you can enter the building that is always all the way at the far right end of town. Inside is a town boss, the source of the vampire scourge (for this area, anyhow). Teach him a lesson (\"Don't think. Feeeeeeelll!\") and you'll enjoy being toadied to by your sidekick. Then the happy mayor will give you a password and you're on your way to the next town. VI. What the hell ARE these things I'm fighting? Those, gamer, are hopping vampires -- China's version of the ubiquitous super- natural parasitic entity that has frightened many a superstitious person for hundreds of years. I believe the game's term for them -- \"Kyonshi\" -- is Mandarin Chinese for \"hopping vampire\". However, it could be a bastardized Chinese put into a game for mainly Japanese players. I think in Cantonese the term is \"Jiangshi\". I'll refer to them throughout this document as \"Kyonshi\", simply because that's what the game calls them. What makes these guys different from their Romanian \"cousins\"? Well, their origins are slightly different. Kyonshi are \"undead\", as in those who have died, but somehow have had their souls trapped in their body instead of escaping to the Nine Heavens (or Hells) after death. This could be for any number of reasons; bad chi flow through the house, suicide, lots of angst, etc. Anyhow, because they have died, their bodies aren't alive, they're just \"undead\". This means that they're not as limber as they once were, hence the hopping; it's the only way they can get around. What motivates them? Not much. Unlike the Western vampire, a Kyonshi has little or no conscious thought. Its only thought is to attack living things and eat their bodies. Since they don't consume their victims whole their victims become vampires as well. Kyonshi don't speak, don't respond intelligently to problems. This is a noticable advantage that the phantom fighter has over his unnatural opponents. A nimble phantom fighter should dance about to keep his assailant's long fingernails (so sharp they're almost claws) at bay. For those interested in Kyonshi I'll provide a bibliography at the end of this FAQ -- they're pretty cool. It also helps to know something about them for later (see below). VII. Do the towns follow a basic pattern? Yes. For the most part they are a bunch of houses, a temple, a \"boss shack\", and a martial arts studio. You can always recognize the temple and the studio, they're the same in every town. The temple is a large building with pillars in the front and lots of pillars and Buddha statues inside. The studio is a long, one-level wooden building. There will be three places where you can get gems as well, and the rest of the little shacks round out the town with places to kill Kyonshi and get ancient scrolls. This seems as good a time as any to explain what the little messages you get when you press A or B in front of a house mean. \"Kyonshies are here\": Ahem. \"Kyonshies are not here\":Ahem. Ahem. Okay, besides the obvious, these mean there is just a basic bunch of Kyonshies in the house (or the house is empty). Fight through all of these and you'll get some ancient scrolls. Empty houses seem to have no point; you walk all the way to the end and some cowering peasant thanks you or says something worthy of the term \"non sequitir.\" \"This is a temple.\": What the Christ. It's a temple. \"There's danger in the air\": This means that inside is either a sacred item or one of the three gems you need to bust them ghosts. \"What, nothing's happening?\":or something like that. Basically one of those houses with goodies or gems in them that you've cleared out already. \"Open the sealenter\": Obviously an error in the programming, this should read something like \"Open the seal to enter\". This is the \"boss shack\" that you need the gems to get inside. \"Enter with courage\":When you have the gems, you can open the seal to the \"boss shack\" and instruct the Kyonshi on the finer points of kung fu. (\"I hope you were paying attention.\") VIII. From the looks of things, I'm not a very good phantom fighter. I just got my butt kicked. Any way to improve on this sad situation? Fortunately, yes. At the beginning it may seem like you'll have a lame kick and punch for the entire game, but fear not. You can improve to such powerful kung-fu that you'll easily steamroll enemies. So, how do you go about improving your skills? Easy. First, you have to do a little struggling; you need to earn ancient scrolls. Just pick a small house and go in. When you get some scrolls go to the studio. When you enter, a fat guard accosts you, and asks you if you understand that you won't get kung-fu lessons for free (This is basically to spare you from the annoying un-skippable dialog that follows if you don't want to enter after all). When you say that you understand, he asks you a question, to test your knowledge. A. What the hell is this? Copy protection? I have no idea why this in the game. I can only guess that at the time this was made, it made sense, or really was to protect from people pirating the game. I don't have the manual, so I don't know. Don't sweat it too much, the questions are pretty easy, often it's obvious what the correct answer is, and if you fail you can always try again. B. Fuck that. Just give me the questions and answers. Ok, I guess quiz games aren't your thing. Here are some of the questions that the fat guy asks, and their answers: (These are from memory, more to come) \"Name an FCI video game.\" I think all of these are FCI games, pick \"WCW\" just to be sure. \"Why do Kyonshies only come out at night?\" They hate the sun, obviously. \"How do you capture a Kyonshie?\" Although you never see it in the game, you use an Urn. \"What is a Kyonshies least favorite food?\" I don't know where this hell this came from, by process of elimination it's ice cream. When you answer correctly the fat guy lets you pass and you can enter the Master's chambers, where they gather for the feast. And though they stab it with their steely knives, they just can't kill the beast. Sorry. If you got that reference you'll know the special sort of anguish I feel now at selling my copy of that album. Anyhow, the master will train you in kung fu in exchange for ancient scrolls. Why, I don't know, but my motto is do what works. So, cough up scrolls and train in the martial arts. The different moves you can select are listed with their scroll cost to the left of them. X. What are the different kung-fu moves, how much do they cost, and what do they do? Good question, grasshopper. There are three basic move categories: \"moves\" (\"mv\"):your movement \"thrust\" :punching moves \"kick\" :ahem, kicking moves. Here's a breakdown by category: MOVES: Wolf Move (\"wolf mv.\") Cost: 2 scrolls What it does: Makes you move faster. Useful. Tiger Move (\"tiger mv.\") Cost: 6 scrolls What it does: see above. Mirage Move (\"mirage mv.\") Cost: 6 scrolls What it does: see above. Dragon Move (\"dragon mv.\") Cost: 18 scrolls What it does: see above. Mirage Walk (\"mirage wk.\") Cost: x scrolls What it does: Allows you to walk while crouching. Mirage Thrust (\"mirage th.\") Cost: 50 scrolls What it does: Allows you to punch while crouching. A terrific move. High Jump Cost: x scrolls What it does: you jump higher. Absolutely necessary for airborne vampires. Windmill Jump (\"wind jump\") Cost: x scrolls What it does: you jump still higher, and you flip in the air as you do. Allows you full range with airborne enemies and looks really cool. THRUSTS: 2 Thrust Cost: 2 scrolls What it does: Gives you two lightning-fast punches instead of one. 3 Thrust Cost: x scrolls What it does: Basically same as above, except now it's a 1-2-3 punch. 4 Thrust Cost: 90 scrolls What it does: Now, you punch both high and low, twice. The most powerful move in the game, believe it or not. KICKS: 2 Kick Cost: 6 scrolls What it does: the 1-2 kick. One for both ears. :) Side kick Cost: 18 scrolls What it does: you lean to the side and use your hips to put a little torque action into your kick. Good damage, another move you'll use right up to the end. Windmill Kick (\"wind kick\") Cost: 50 scrolls What it does: When you jump straight up in the air and kick, you'll spin around in a flurry of feet. Really only useful for airborne enemies, and doesn't always come off clean. Jump Kick Cost: 80 scrolls What it does: Run at your enemy and kick and you'll leap at them with both feet forward. (\"Mind your Manners!!!\") Great as an opening move, as well as for enemies who like to jump a lot. XI. What are some good fighting tactics? Stick and move. If you stand in one spot you'll get creamed, as more often than not enemies will soak one or two kicks and swipe you with them claws. This is most important at the beginning of the game, when you can't do anything but punch and kick, and move around a little bit. The main method of attack Kyonshies use is to take little hops at you, then when they get close enough to a big Kyonshie leap into your face. What you want to try to do is to place yourself at a point where the Kyonshie will be in midleap when it comes into kick range. Then, when it leaps, kick it, and it'll fall over. It won't die, but knocking it down gives you breathing room. You'll take a lot of hits from Kyonshies just steadily leaping towards you. That's their other fighting tactic...they just keep coming, soaking up the kicks and punches until they can cut you. If you can kick them enough times, however, they'll fall over. Different kyonshies take more or less kicks/punches in a row to be knocked down, size is one factor. Later, when you learn more kung fu, it will be easier to defeat enemies, although the enemies will get progressively harder as you go from town to town. One good move that will help you out a lot is the \"Wind Jump\". This allows you to leap high into the air (doing a cool kung-fu flip) above the reach of the dreaded hoppping demons. When enemies come at you, kick them once or twice, and if they don't fall down hop over them just before they get to you. A little practice helps, but soon you will be able to drop down right on the other side of them and kick them in the back of the head while they are turning around. Kyonshies are very stupid. It helps a lot to remember this, as the next trick shows. See, Kyonshies (as you will know if you read the above section on them) are in the throes of rigor mortis, their bodies in an arrested state of decay. So they can only see straight forward, right? So if you crouch down, they will turn back and forth, their tiny minds trying to figure out how you pulled your disappearing act. This will not be very useful in the beginning, as you can't walk while crouching. DON'T try this just before a Kyonshie gets to you -- their turning back and forth motion will still be able to hit you, as you are not completely under their arm level (unless you are fighting a really tall kyonshie, see below). This move becomes more useful when you learn the Mirage Walk, as you'll be able to shuffle about under their vision, just don't attempt to get too close to them as they can often still hit you. Later, when you learn Mirage Thrust, this becomes a winning tactic, allowing you to stand under tall enemies and bosses and punch them in their unfeeling vampire gonads mercilessly. (WARNING: This is kind of a cheapo move, as you really can kill tall kyonshi bosses with this tactic. It certainly doesn't _look_ very exciting.) XII. Alright. I'm in town. What do I do first? Wander through town until you find the temple, then try to fight your way through it, shouldn't be to difficult. In the first town, it's one of the first buildings, later on it will be farther into town, or sometimes nonexistent. Anyhow, it's important to start here, so that you can fill up your health, you'll need to often and there's only one other way to do it in the game, and that involves more fighting. Once you've done that, go to the kung-fu studio and see how many scrolls you'll need to learn the various kung-fu moves. The dialog trees can be annoying, but get used to them because they are all through the game. :P Anyhow, leave the studio with the moves you want to learn in mind, and find a house with Kyonshies in it. Then enter and work your way to get the scrolls. Surprisingly (or perhaps not, depending on how well versed you are in \"game logic\"), you can enter the same house again and again, continuously obtaining scrolls. You can get up to 99, but don't bother trying that here unless you are sick in bed with chicken pox or something -- you'll have better oppportunities later on when you recieve lots more scrolls for rescuing townspeople. Learn all the moves you can, when the master tells you you've tapped him out, you can move on. A. Just to make it easier on myself, what moves should I learn first? Since your kung-fu is puny, learn attack moves first. Specifically, get 2 Kick and 2 Thrust. Throughout the game, when you have an opportunity to learn an attack move, learn it, and learn the other moves later. While the \"Move\" moves may seem pointless, they are actually very useful, since being able to move quickly will allow you to dance in an out of a kyonshie's range without getting creamed every time you try to hit them. It's also highly important to learn the jump moves, since they allow you to attack airborne enemies, of which there are 3 in the game, all of them important. While I suppose you could beat the game without all of the moves, I wouldn't recommend it, simply because it wouldn't be as fun, as learning new moves is one of the high points of the game. After you've learned all the moves you care to, head for the houses with the gems in them, collect the gems, and head for the boss shack. Don't forget to write down your password! XIII. Hey, these Kyonshies aren't all alike! Quite right. There are several different types of Kyonshies. Basically, there are four basic types of Kyonshi, differentiated by body type: the little one, the medium one, the tall one, and the big fat one. Within these different morphical categories there's lots of room for variation in the different colors they wear. You'll no doubt become quite familiar with the types as you play the game, and certain ones will become infamous. (\"ARgh! Not more green ones!\") XIV. Okay, so there's different types. What are they like? Without further ado: little: This little bastard is fast and hard to hit. He'll be a toughie when you're duking it out in the first town and don't have awesome kung-fu. His basic weakness is that he can't jump that far, and he's really short, making him easy to hop over. Fortunately he's still just tall enough to kick in the face. medium: Your basic vanilla Kyonshi. Some types jump a lot, some are fast, etc. tall: The tall guy moves ponderously slow, and thanks to his height is hard to jump over. He also jumps very high while attacking as well. Naturally enough this is a big mistake on his part; he jumps so high that you can crouch and let him go over you, then \"boot to the head\" while he's turning around. Later on when you learn the Mirage Thrust the tall guy will be at your mercy as your firsts pound mercilessly into his undead loins, and his turning back and forth schtick won't be able to nail you. fat: This guy is tougher than he looks. He's very strong, such that even though he moves slowly, he packs a real wallop. Also, he doesn't jump that high, but is still tall enough that jumping over him is a bit of a sticky wicket. Your big advantage over him is speed. While he can soak up hits and really wummox you when he gets a hit in, if you play it right you can kick once or twice then get the hell out of the way when the Kyonshi express comes through. XV. I won some kind of item, what does it do? Aha, one of the coveted mystical items that you can use in the game. Don't confuse these with scrolls -- ancient scrolls sound cool but don't do jack except buy you kung-fu. However, the items are quite nice. First, here's how to use an item once you've got it. Whenever you obtain an item, you will see it appear at the top of the screen, in a long box to the left of the stands that hold the three gems you'll need to get out of town. However, make no mistake, you are NOT carrying that item. Your faithful sidekick (who has no name that I could tell) is carrying them. Thus, whenever you enter a building, and you want to use an item, turn back as if to leave. Your boy will pop his head in and ask if you want to split, use an item, or ignore him and get back to butt-kicking. Well, ask for an item, and he'll give you a list: strangely enough, he lists all the items you can get, even if you don't have them yet. Pick one, and it will be outlined by a red square at the top of the screen. This means you are now carrying the item, and will use it instead of punching when you push A. Now, here's all the items, and what they do. Talisman: Freezes enemies in their tracks sometimes. Tonten: Knocks baddies onto their cans with a flash of light. Sword: Knocks guys down, you can also slice 'em with it. Bell: Controls the demon-boy. A note of warning: your items will also take hits for you if you get hit. However, this will also break the item, making it useless. Hey, it's an antique. Keep this in mind when fighting with an item. It's a good idea to grab them, they're nice for keeping enemies at bay, and the Tonten is pretty damn invaluable for beating the game. XVI. I bought one of those Game Genies. Any codes for this game? CODE KEY IN . . . EFFECT . . . 1 VTVKEGSA + KAVKOGNA Start with Sword 2 VTVKEGSA + SAVKOGNA Start with Bell 3 VTVKEGSA + UAVKOGNA Start with Tonten 4 VTVKEGSA + XAVKOGNA Start with Talisman 5 LASKNGAA + VAVKOGNA Start with 3 Scrolls 6 TASKNGAA + VAVKOGNA Start with 6 Scrolls 7 SXSZLUSE Infinite energy 8 OVSZPLSV + PESZZLAA Take less damage when attacked (author's note: This is taken verbatim from Galoob's master list of all Game Genie codes for NES games that they ever published. There may, of course, be some home- grown codes floating about. So all due credit goes to Galoob for the above blurb.) XVII. This isn't the most complete FAQ I've ever read, you know... Sorry. This is mostly from memory. If I get e-mails requesting updates I'll saunter back into the game with my rightfully earned cheat codes and research it fully for those who desire it. And besides, if you think this FAQ is incomplete, check out some of the other (admittedly) sparse fare at TSR's NES FAQ page. For right now I'll call an end to this FAQ, just because I figure this is enough to be written about any NES game that wasn't made by Square. :) I'll send my regards first and foremost to TSR, who's NES page is truly one of the best place for NES info on the 'net, not to mention skilled use of graphics and delightful content like the NES oddity page. Also, here's to the place that posted the ROM I used for this FAQ, Big Daddy's International House of ROMs, THE place to get ROMs on the 'net. In my experience, if he doesn't have it here, it doesn't exist yet (at least for NES, anyhow). I'll also send out greets to John Turk, who's NES Underground Library is probably the most ambitious of all the NES pages I've seen. And finally I'll send shots out to all the other NES pages, all those who've stayed cool despite NESticle's appearance, and to all the fellows on IRC channels like #emuroms, #emulator, and #emu who are out there circulating coolness in less than 200k. :) Also a shot out to my friends on #1980Warez, who have nothing to do with NES ROMs, but are cool anyway. Jibes go to all those who feel the need to charge excessive amounts of money for their emulators. Say what you will about effort and reward, guys, but in the end you're creating something used to do something that's, shall we say, a trifle off-color in the eyes of the law. Jibes also go to those who run IRC channels so huge that no chatting goes on in them, and channels that have nothing but DCC bots in them. Thanks for helping turn IRC into a wasteland, guys. So much have I written for gamers, now give me a drink! The Deacon Restore Page","title":"Phantom Fighter FAQ"},{"location":"nes/#training-hall-quiz-questions-and-answers","text":"+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^+ < ////// // // /// //// // ////// ///// ////////// > < // // // // // // // // // // // // // // // > < ///// ////// /////// // //// // // // // // // > < // // // // // // /// // // // // // // > < // // // // // // // // ///// // // // > < > < ////// // ///// // // ////// ////// ////// > < // // // // // // // // // > < //// // // /// ////// // //// //// > < // // // // // // // // // // > < // // ///// // // // ////// // // > +------------------------------------------------------------------+ * * * * * * Training Hall Quiz Questions and Answers * * * * * * A guide by PinKirby (Pinku_Kirby@yahoo.com) +-----------------+ ----Version 1.0---- +-----------------+ June 9th, 2004 -Began this guide -------------------- Contents: 1. Introduction 2. The Training Hall 3. The Questions and Answers 4. Possible Messages 5. Thanks 6. Legal Info --------------------- * * * * * * 1. Introduction * * * * * * Phantom Fighter is a game that is not very well-known, but is still, in my opinion, a great game. I've decided to make a guide on the questions the guard of the Training Hall asks you. Most questions have one correct answer, but some questions are asked in which ALL of the multiple choices are correct answers. Here's how to use this guide, using a sample question NOT asked in the game: ***QUESTION: What game does this guide cover? 1. Super Mario Brothers (X) 2. Phantom Fighter (O) 3. This game has no name(X) (X) - Means an incorrect answer, in which the guard throws you out. (O) - Means a correct answer, in which the guard lets you pass. --------------------------------------------------------------- * * * * * * 2. The Training Hall * * * * * * After defeating the monsters, the sharp-nailed Chinese phantoms which are known as \"Kyonshies\", in houses and buildings, you can get one or more scrolls. You use these scrolls to learn skills in the Training Hall, but you can't see the master for free, as his guard explains to you. To prove you are worthy of the master's time and attention, you must correctly answer a question with one of three multiple choices. While many of the guard's questions pertain to your foes the Kyonshies, some of the questions involve trivia. If you get a question wrong, DON'T FRET! You can re-enter and try again with another question. The guard tells you, Kenchi: \"You can't see the master for free. Do you understand?\" If you answer no, he'll laugh at your cowardice and kick you out. Answer yes, and it's quiz time! --------------------------------------------------------------- * * * * * * 3. The Questions and Answers * * * * * * QUESTION 1: How many stars are there in the American flag? 1. 50 (O) 2. 28 (X) 3. 5 (X) --------------------- QUESTION 2: What country do samurais come from? 1. India (X) 2. Japan (O) 3. Spain (X) --------------------- QUESTION 3: What is the best method to make sure Kyonshies never revive again? 1. Big Stakes (X) 2. Seal in Rock (X) 3. Fry in Oil (O) --------------------- QUESTION 4: What is the teaching taught by Confucius called? 1. Psychiatry (X) 2. Confucianism (O) 3. Physiognomy (X) --------------------- QUESTION 5: Who built the Great Wall of China? 1. The Emperor (O) 2. Plasterer (O) 3. Farmer (O) --------------------- QUESTION 6: What is Kyonshies' most powerful weapon? 1. Sharp Claws (O) 2. Sharp Eyes (X) 3. Swift Moves (X) --------------------- QUESTION 7: What is the horrible skill called when your head is banged against your opponent's head? 1. Head Strike (O) 2. Head batting (O) 3. Head to Head (O) --------------------- QUESTION 8: What's the name of George Bush's dog? (This is George Bush Sr.!) 1. Millie (O) 2. Martha (X) 3. Pooch (X) --------------------- QUESTION 9: Who created Ultima? 1. Lord British (O) 2. Prince Chuck (X) 3. Mr. Exodus (X) --------------------- QUESTION 10: What is the best thing to use to capture Kyonshies? 1. Coffin (X) 2. Urn (O) 3. Jewelry Box (X) --------------------- QUESTION 11: What is the famous Chinese Newspaper? 1. School Paper (X) 2. Wall Poster (O) 3. Daily China (X) --------------------- QUESTION 11: What is the food Kyonshies hate? 1. Liver (X) 2. Ice Cream (O) 3. Frog Eyes (X) --------------------- QUESTION 12: Something is used to beat Kyonshies. What is it? 1. Bird's Blood (O) 2. Lizard Scale (X) 3. Vulture Nail (X) --------------------- QUESTION 13: Why do Kyonshies only come out at night? 1. Play late (X) 2. Shy (X) 3. Hate the Sun (O) --------------------- QUESTION 14: Which emperor is often called the Last Emperor? 1. Sagi (X) 2. Higi (X) 3. Fugi (O) --------------------- QUESTION 15: What kind of place do Kyoshies usually live in? 1. Wet place (O) 2. New Jersey (O) 3. Beverly Hill (O) --------------------- QUESTION 16: What is the skill called when you hold your opponent's arms from the back and throw him backward? 1. Dragon Suplex (O) 2. Front Suplex (X) 3. Side Suplex (X) --------------------- QUESTION 17: Name an FCI Video Game. 1. Hydlide (O) 2. WCW (O) 3. Ultima (O) --------------------- QUESTION 18: What is a Chinese martial art usually called? 1. Kung Fu (O) 2. Martial Art (X) 3. Phantom Kick (X) --------------------- QUESTION 19: What is another word for one? 1. Uno (O) 2. Ichi (O) 3. Un (O) --------------------- --------------------------------------------------------------- * * * * * * 4. Possible Messages * * * * * * ----- CORRECT MESSAGES: - That's good. The master is waiting for you. Go and see him. If you answer \"New Jersey\" or \"Beverly Hill\" to the question about where Kyonshies usually live in: - That's right. Now as you know, that is the hangout of Kyonshi. Go and enter. (I'm assuming the guard pities you so much that he just accepts it as a correct answer. :P ) ----- INCORRECT MESSAGES: - Why are you trying to pretend you don't know the answer? - You, Kenchi, not know a thing like this? Worth a laugh! Ha ha ha! Now, get out! Get out! - That was a pity. Train yourself and come again. - How little you know I'm stunned. Think it over and come back again. - Kenchi, you are really an idiot! Go home, try to learn more, and come back when you are wiser. If you answer \"5\" as to how many stars are on the US flag: - Dunce! --------------------------------------------------------------- * * * * * * 5. Thanks * * * * * * - To my sister, Mom and Grandpa - To FCI and Pony Canyon for making this game - To YOU for taking the time to read this guide! * * * * * * 6. Legal Stuff * * * * * * This guide is copyrighted (c) by PinKirby. You may copy SOME of this guide, but you MUST give me, PinKirby, credit. If you want to put this guide up on your website, email me and I will see about it. If I find that this guide is on a website NOT listed below, then I will have it removed. SITES THAT MAY HAVE THIS GUIDE UP: www.GameFaqs.com Phatom Fighter is (c) 1989 by FCI and Pony Canyon. The Nintendo Entertainment System and Nintendo of America are registered trademarks of Nintendo. All rights reserved. This Phantom Fighter Training Hall Quiz Guide is (c) by PinKirby Restore Page","title":"Training Hall Quiz Questions and Answers"},{"location":"ollama/","text":"API Endpoints Generate a completion Generate a chat completion Create a Model List Local Models Show Model Information Copy a Model Delete a Model Pull a Model Push a Model Generate Embeddings Conventions Model names Model names follow a model:tag format, where model can have an optional namespace such as example/model . Some examples are orca-mini:3b-q4_1 and llama2:70b . The tag is optional and, if not provided, will default to latest . The tag is used to identify a specific version. Durations All durations are returned in nanoseconds. Streaming responses Certain endpoints stream responses as JSON objects and can optional return non-streamed responses. Generate a completion POST /api/generate Generate a response for a given prompt with a provided model. This is a streaming endpoint, so there will be a series of responses. The final response object will include statistics and additional data from the request. Parameters model : (required) the model name prompt : the prompt to generate a response for images : (optional) a list of base64-encoded images (for multimodal models such as llava ) Advanced parameters (optional): format : the format to return a response in. Currently the only accepted value is json options : additional model parameters listed in the documentation for the Modelfile such as temperature system : system message to (overrides what is defined in the Modelfile ) template : the prompt template to use (overrides what is defined in the Modelfile ) context : the context parameter returned from a previous request to /generate , this can be used to keep a short conversational memory stream : if false the response will be returned as a single response object, rather than a stream of objects raw : if true no formatting will be applied to the prompt. You may choose to use the raw parameter if you are specifying a full templated prompt in your request to the API keep_alive : controls how long the model will stay loaded into memory following the request (default: 5m ) JSON mode Enable JSON mode by setting the format parameter to json . This will structure the response as a valid JSON object. See the JSON mode example below. Note: it's important to instruct the model to use JSON in the prompt . Otherwise, the model may generate large amounts whitespace. Examples Generate request (Streaming) Request curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\", \"prompt\": \"Why is the sky blue?\" }' Response A stream of JSON objects is returned: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T08:52:19.385406455-07:00\", \"response\": \"The\", \"done\": false } The final response in the stream also includes additional data about the generation: total_duration : time spent generating the response load_duration : time spent in nanoseconds loading the model prompt_eval_count : number of tokens in the prompt prompt_eval_duration : time spent in nanoseconds evaluating the prompt eval_count : number of tokens the response eval_duration : time in nanoseconds spent generating the response context : an encoding of the conversation used in this response, this can be sent in the next request to keep a conversational memory response : empty if the response was streamed, if not streamed, this will contain the full response To calculate how fast the response is generated in tokens per second (token/s), divide eval_count / eval_duration . { \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"response\": \"\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 10706818083, \"load_duration\": 6338219291, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 130079000, \"eval_count\": 259, \"eval_duration\": 4232710000 } Request (No streaming) Request A response can be received in one reply when streaming is off. curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\", \"prompt\": \"Why is the sky blue?\", \"stream\": false }' Response If stream is set to false , the response will be a single JSON object: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"response\": \"The sky is blue because it is the color of the sky.\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 5043500667, \"load_duration\": 5025959, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 325953000, \"eval_count\": 290, \"eval_duration\": 4709213000 } Request (JSON mode) When format is set to json , the output will always be a well-formed JSON object. It's important to also instruct the model to respond in JSON. Request curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\", \"prompt\": \"What color is the sky at different times of the day? Respond using JSON\", \"format\": \"json\", \"stream\": false }' Response { \"model\": \"llama2\", \"created_at\": \"2023-11-09T21:07:55.186497Z\", \"response\": \"{\\n\\\"morning\\\": {\\n\\\"color\\\": \\\"blue\\\"\\n},\\n\\\"noon\\\": {\\n\\\"color\\\": \\\"blue-gray\\\"\\n},\\n\\\"afternoon\\\": {\\n\\\"color\\\": \\\"warm gray\\\"\\n},\\n\\\"evening\\\": {\\n\\\"color\\\": \\\"orange\\\"\\n}\\n}\\n\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 4648158584, \"load_duration\": 4071084, \"prompt_eval_count\": 36, \"prompt_eval_duration\": 439038000, \"eval_count\": 180, \"eval_duration\": 4196918000 } The value of response will be a string containing JSON similar to: { \"morning\": { \"color\": \"blue\" }, \"noon\": { \"color\": \"blue-gray\" }, \"afternoon\": { \"color\": \"warm gray\" }, \"evening\": { \"color\": \"orange\" } } Request (with images) To submit images to multimodal models such as llava or bakllava , provide a list of base64-encoded images : Request curl http://localhost:11434/api/generate -d '{ \"model\": \"llava\", \"prompt\":\"What is in this picture?\", \"stream\": false, \"images\": [\"iVBORw0KGgoAAAANSUhEUgAAAG0AAABmCAYAAADBPx+VAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAA3VSURBVHgB7Z27r0zdG8fX743i1bi1ikMoFMQloXRpKFFIqI7LH4BEQ+NWIkjQuSWCRIEoULk0gsK1kCBI0IhrQVT7tz/7zZo888yz1r7MnDl7z5xvsjkzs2fP3uu71nNfa7lkAsm7d++Sffv2JbNmzUqcc8m0adOSzZs3Z+/XES4ZckAWJEGWPiCxjsQNLWmQsWjRIpMseaxcuTKpG/7HP27I8P79e7dq1ars/yL4/v27S0ejqwv+cUOGEGGpKHR37tzJCEpHV9tnT58+dXXCJDdECBE2Ojrqjh071hpNECjx4cMHVycM1Uhbv359B2F79+51586daxN/+pyRkRFXKyRDAqxEp4yMlDDzXG1NPnnyJKkThoK0VFd1ELZu3TrzXKxKfW7dMBQ6bcuWLW2v0VlHjx41z717927ba22U9APcw7Nnz1oGEPeL3m3p2mTAYYnFmMOMXybPPXv2bNIPpFZr1NHn4HMw0KRBjg9NuRw95s8PEcz/6DZELQd/09C9QGq5RsmSRybqkwHGjh07OsJSsYYm3ijPpyHzoiacg35MLdDSIS/O1yM778jOTwYUkKNHWUzUWaOsylE00MyI0fcnOwIdjvtNdW/HZwNLGg+sR1kMepSNJXmIwxBZiG8tDTpEZzKg0GItNsosY8USkxDhD0Rinuiko2gfL/RbiD2LZAjU9zKQJj8RDR0vJBR1/Phx9+PHj9Z7REF4nTZkxzX4LCXHrV271qXkBAPGfP/atWvu/PnzHe4C97F48eIsRLZ9+3a3f/9+87dwP1JxaF7/3r17ba+5l4EcaVo0lj3SBq5kGTJSQmLWMjgYNei2GPT1MuMqGTDEFHzeQSP2wi/jGnkmPJ/nhccs44jvDAxpVcxnq0F6eT8h4ni/iIWpR5lPyA6ETkNXoSukvpJAD3AsXLiwpZs49+fPn5ke4j10TqYvegSfn0OnafC+Tv9ooA/JPkgQysqQNBzagXY55nO/oa1F7qvIPWkRL12WRpMWUvpVDYmxAPehxWSe8ZEXL20sadYIozfmNch4QJPAfeJgW3rNsnzphBKNJM2KKODo1rVOMRYik5ETy3ix4qWNI81qAAirizgMIc+yhTytx0JWZuNI03qsrgWlGtwjoS9XwgUhWGyhUaRZZQNNIEwCiXD16tXcAHUs79co0vSD8rrJCIW98pzvxpAWyyo3HYwqS0+H0BjStClcZJT5coMm6D2LOF8TolGJtK9fvyZpyiC5ePFi9nc/oJU4eiEP0jVoAnHa9wyJycITMP78+eMeP37sXrx44d6+fdt6f82aNdkx1pg9e3Zb5W+RSRE+n+VjksQWifvVaTKFhn5O8my63K8Qabdv33b379/PiAP//vuvW7BggZszZ072/+TJk91YgkafPn166zXB1rQHFvouAWHq9z3SEevSUerqCn2/dDCeta2jxYbr69evk4MHDyY7d+7MjhMnTiTPnz9Pfv/+nfQT2ggpO2dMF8cghuoM7Ygj5iWCqRlGFml0QC/ftGmTmzt3rmsaKDsgBSPh0/8yPeLLBihLkOKJc0jp8H8vUzcxIA1k6QJ/c78tWEyj5P3o4u9+jywNPdJi5rAH9x0KHcl4Hg570eQp3+vHXGyrmEeigzQsQsjavXt38ujRo44LQuDDhw+TW7duRS1HGgMxhNXHgflaNTOsHyKvHK5Ijo2jbFjJBQK9YwFd6RVMzfgRBmEfP37suBBm/p49e1qjEP2mwTViNRo0VJWH1deMXcNK08uUjVUu7s/zRaL+oLNxz1bpANco4npUgX4G2eFbpDFyQoQxojBCpEGSytmOH8qrH5Q9vuzD6ofQylkCUmh8DBAr+q8JCyVNtWQIidKQE9wNtLSQnS4jDSsxNHogzFuQBw4cyM61UKVsjfr3ooBkPSqqQHesUPWVtzi9/vQi1T+rJj7WiTz4Pt/l3LxUkr5P2VYZaZ4URpsE+st/dujQoaBBYokbrz/8TJNQYLSonrPS9kUaSkPeZyj1AWSj+d+VBoy1pIWVNed8P0Ll/ee5HdGRhrHhR5GGN0r4LGZBaj8oFDJitBTJzIZgFcmU0Y8ytWMZMzJOaXUSrUs5RxKnrxmbb5YXO9VGUhtpXldhEUogFr3IzIsvlpmdosVcGVGXFWp2oU9kLFL3dEkSz6NHEY1sjSRdIuDFWEhd8KxFqsRi1uM/nz9/zpxnwlESONdg6dKlbsaMGS4EHFHtjFIDHwKOo46l4TxSuxgDzi+rE2jg+BaFruOX4HXa0Nnf1lwAPufZeF8/r6zD97WK2qFnGjBxTw5qNGPxT+5T/r7/7RawFC3j4vTp09koCxkeHjqbHJqArmH5UrFKKksnxrK7FuRIs8STfBZv+luugXZ2pR/pP9Ois4z+TiMzUUkUjD0iEi1fzX8GmXyuxUBRcaUfykV0YZnlJGKQpOiGB76x5GeWkWWJc3mOrK6S7xdND+W5N6XyaRgtWJFe13GkaZnKOsYqGdOVVVbGupsyA/l7emTLHi7vwTdirNEt0qxnzAvBFcnQF16xh/TMpUuXHDowhlA9vQVraQhkudRdzOnK+04ZSP3DUhVSP61YsaLtd/ks7ZgtPcXqPqEafHkdqa84X6aCeL7YWlv6edGFHb+ZFICPlljHhg0bKuk0CSvVznWsotRu433alNdFrqG45ejoaPCaUkWERpLXjzFL2Rpllp7PJU2a/v7Ab8N05/9t27Z16KUqoFGsxnI9EosS2niSYg9SpU6B4JgTrvVW1flt1sT+0ADIJU2maXzcUTraGCRaL1Wp9rUMk16PMom8QhruxzvZIegJjFU7LLCePfS8uaQdPny4jTTL0dbee5mYokQsXTIWNY46kuMbnt8Kmec+LGWtOVIl9cT1rCB0V8WqkjAsRwta93TbwNYoGKsUSChN44lgBNCoHLHzquYKrU6qZ8lolCIN0Rh6cP0Q3U6I6IXILYOQI513hJaSKAorFpuHXJNfVlpRtmYBk1Su1obZr5dnKAO+L10Hrj3WZW+E3qh6IszE37F6EB+68mGpvKm4eb9bFrlzrok7fvr0Kfv727dvWRmdVTJHw0qiiCUSZ6wCK+7XL/AcsgNyL74DQQ730sv78Su7+t/A36MdY0sW5o40ahslXr58aZ5HtZB8GH64m9EmMZ7FpYw4T6QnrZfgenrhFxaSiSGXtPnz57e9TkNZLvTjeqhr734CNtrK41L40sUQckmj1lGKQ0rC37x544r8eNXRpnVE3ZZY7zXo8NomiO0ZUCj2uHz58rbXoZ6gc0uA+F6ZeKS/jhRDUq8MKrTho9fEkihMmhxtBI1DxKFY9XLpVcSkfoi8JGnToZO5sU5aiDQIW716ddt7ZLYtMQlhECdBGXZZMWldY5BHm5xgAroWj4C0hbYkSc/jBmggIrXJWlZM6pSETsEPGqZOndr2uuuR5rF169a2HoHPdurUKZM4CO1WTPqaDaAd+GFGKdIQkxAn9RuEWcTRyN2KSUgiSgF5aWzPTeA/lN5rZubMmR2bE4SIC4nJoltgAV/dVefZm72AtctUCJU2CMJ327hxY9t7EHbkyJFseq+EJSY16RPo3Dkq1kkr7+q0bNmyDuLQcZBEPYmHVdOBiJyIlrRDq41YPWfXOxUysi5fvtyaj+2BpcnsUV/oSoEMOk2CQGlr4ckhBwaetBhjCwH0ZHtJROPJkyc7UjcYLDjmrH7ADTEBXFfOYmB0k9oYBOjJ8b4aOYSe7QkKcYhFlq3QYLQhSidNmtS2RATwy8YOM3EQJsUjKiaWZ+vZToUQgzhkHXudb/PW5YMHD9yZM2faPsMwoc7RciYJXbGuBqJ1UIGKKLv915jsvgtJxCZDubdXr165mzdvtr1Hz5LONA8jrUwKPqsmVesKa49S3Q4WxmRPUEYdTjgiUcfUwLx589ySJUva3oMkP6IYddq6HMS4o55xBJBUeRjzfa4Zdeg56QZ43LhxoyPo7Lf1kNt7oO8wWAbNwaYjIv5lhyS7kRf96dvm5Jah8vfvX3flyhX35cuX6HfzFHOToS1H4BenCaHvO8pr8iDuwoUL7tevX+b5ZdbBair0xkFIlFDlW4ZknEClsp/TzXyAKVOmmHWFVSbDNw1l1+4f90U6IY/q4V27dpnE9bJ+v87QEydjqx/UamVVPRG+mwkNTYN+9tjkwzEx+atCm/X9WvWtDtAb68Wy9LXa1UmvCDDIpPkyOQ5ZwSzJ4jMrvFcr0rSjOUh+GcT4LSg5ugkW1Io0/SCDQBojh0hPlaJdah+tkVYrnTZowP8iq1F1TgMBBauufyB33x1v+NWFYmT5KmppgHC+NkAgbmRkpD3yn9QIseXymoTQFGQmIOKTxiZIWpvAatenVqRVXf2nTrAWMsPnKrMZHz6bJq5jvce6QK8J1cQNgKxlJapMPdZSR64/UivS9NztpkVEdKcrs5alhhWP9NeqlfWopzhZScI6QxseegZRGeg5a8C3Re1Mfl1ScP36ddcUaMuv24iOJtz7sbUjTS4qBvKmstYJoUauiuD3k5qhyr7QdUHMeCgLa1Ear9NquemdXgmum4fvJ6w1lqsuDhNrg1qSpleJK7K3TF0Q2jSd94uSZ60kK1e3qyVpQK6PVWXp2/FC3mp6jBhKKOiY2h3gtUV64TWM6wDETRPLDfSakXmH3w8g9Jlug8ZtTt4kVF0kLUYYmCCtD/DrQ5YhMGbA9L3ucdjh0y8kOHW5gU/VEEmJTcL4Pz/f7mgoAbYkAAAAAElFTkSuQmCC\"] }' Response { \"model\": \"llava\", \"created_at\": \"2023-11-03T15:36:02.583064Z\", \"response\": \"A happy cartoon character, which is cute and cheerful.\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 2938432250, \"load_duration\": 2559292, \"prompt_eval_count\": 1, \"prompt_eval_duration\": 2195557000, \"eval_count\": 44, \"eval_duration\": 736432000 } Request (Raw Mode) In some cases, you may wish to bypass the templating system and provide a full prompt. In this case, you can use the raw parameter to disable templating. Also note that raw mode will not return a context. Request curl http://localhost:11434/api/generate -d '{ \"model\": \"mistral\", \"prompt\": \"[INST] why is the sky blue? [/INST]\", \"raw\": true, \"stream\": false }' Request (Reproducible outputs) For reproducible outputs, set temperature to 0 and seed to a number: Request curl http://localhost:11434/api/generate -d '{ \"model\": \"mistral\", \"prompt\": \"Why is the sky blue?\", \"options\": { \"seed\": 123, \"temperature\": 0 } }' Response { \"model\": \"mistral\", \"created_at\": \"2023-11-03T15:36:02.583064Z\", \"response\": \" The sky appears blue because of a phenomenon called Rayleigh scattering.\", \"done\": true, \"total_duration\": 8493852375, \"load_duration\": 6589624375, \"prompt_eval_count\": 14, \"prompt_eval_duration\": 119039000, \"eval_count\": 110, \"eval_duration\": 1779061000 } Generate request (With options) If you want to set custom options for the model at runtime rather than in the Modelfile, you can do so with the options parameter. This example sets every available option, but you can set any of them individually and omit the ones you do not want to override. Request curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\", \"prompt\": \"Why is the sky blue?\", \"stream\": false, \"options\": { \"num_keep\": 5, \"seed\": 42, \"num_predict\": 100, \"top_k\": 20, \"top_p\": 0.9, \"tfs_z\": 0.5, \"typical_p\": 0.7, \"repeat_last_n\": 33, \"temperature\": 0.8, \"repeat_penalty\": 1.2, \"presence_penalty\": 1.5, \"frequency_penalty\": 1.0, \"mirostat\": 1, \"mirostat_tau\": 0.8, \"mirostat_eta\": 0.6, \"penalize_newline\": true, \"stop\": [\"\\n\", \"user:\"], \"numa\": false, \"num_ctx\": 1024, \"num_batch\": 2, \"num_gqa\": 1, \"num_gpu\": 1, \"main_gpu\": 0, \"low_vram\": false, \"f16_kv\": true, \"vocab_only\": false, \"use_mmap\": true, \"use_mlock\": false, \"rope_frequency_base\": 1.1, \"rope_frequency_scale\": 0.8, \"num_thread\": 8 } }' Response { \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"response\": \"The sky is blue because it is the color of the sky.\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 4935886791, \"load_duration\": 534986708, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 107345000, \"eval_count\": 237, \"eval_duration\": 4289432000 } Load a model If an empty prompt is provided, the model will be loaded into memory. Request curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\" }' Response A single JSON object is returned: { \"model\": \"llama2\", \"created_at\": \"2023-12-18T19:52:07.071755Z\", \"response\": \"\", \"done\": true } Generate a chat completion POST /api/chat Generate the next message in a chat with a provided model. This is a streaming endpoint, so there will be a series of responses. Streaming can be disabled using \"stream\": false . The final response object will include statistics and additional data from the request. Parameters model : (required) the model name messages : the messages of the chat, this can be used to keep a chat memory The message object has the following fields: role : the role of the message, either system , user or assistant content : the content of the message images (optional): a list of images to include in the message (for multimodal models such as llava ) Advanced parameters (optional): format : the format to return a response in. Currently the only accepted value is json options : additional model parameters listed in the documentation for the Modelfile such as temperature template : the prompt template to use (overrides what is defined in the Modelfile ) stream : if false the response will be returned as a single response object, rather than a stream of objects keep_alive : controls how long the model will stay loaded into memory following the request (default: 5m ) Examples Chat Request (Streaming) Request Send a chat message with a streaming response. curl http://localhost:11434/api/chat -d '{ \"model\": \"llama2\", \"messages\": [ { \"role\": \"user\", \"content\": \"why is the sky blue?\" } ] }' Response A stream of JSON objects is returned: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T08:52:19.385406455-07:00\", \"message\": { \"role\": \"assistant\", \"content\": \"The\", \"images\": null }, \"done\": false } Final response: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"done\": true, \"total_duration\": 4883583458, \"load_duration\": 1334875, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 342546000, \"eval_count\": 282, \"eval_duration\": 4535599000 } Chat request (No streaming) Request curl http://localhost:11434/api/chat -d '{ \"model\": \"llama2\", \"messages\": [ { \"role\": \"user\", \"content\": \"why is the sky blue?\" } ], \"stream\": false }' Response { \"model\": \"registry.ollama.ai/library/llama2:latest\", \"created_at\": \"2023-12-12T14:13:43.416799Z\", \"message\": { \"role\": \"assistant\", \"content\": \"Hello! How are you today?\" }, \"done\": true, \"total_duration\": 5191566416, \"load_duration\": 2154458, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 383809000, \"eval_count\": 298, \"eval_duration\": 4799921000 } Chat request (With History) Send a chat message with a conversation history. You can use this same approach to start the conversation using multi-shot or chain-of-thought prompting. Request curl http://localhost:11434/api/chat -d '{ \"model\": \"llama2\", \"messages\": [ { \"role\": \"user\", \"content\": \"why is the sky blue?\" }, { \"role\": \"assistant\", \"content\": \"due to rayleigh scattering.\" }, { \"role\": \"user\", \"content\": \"how is that different than mie scattering?\" } ] }' Response A stream of JSON objects is returned: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T08:52:19.385406455-07:00\", \"message\": { \"role\": \"assistant\", \"content\": \"The\" }, \"done\": false } Final response: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"done\": true, \"total_duration\": 8113331500, \"load_duration\": 6396458, \"prompt_eval_count\": 61, \"prompt_eval_duration\": 398801000, \"eval_count\": 468, \"eval_duration\": 7701267000 } Chat request (with images) Request Send a chat message with a conversation history. curl http://localhost:11434/api/chat -d '{ \"model\": \"llava\", \"messages\": [ { \"role\": \"user\", \"content\": \"what is in this image?\", \"images\": [\"iVBORw0KGgoAAAANSUhEUgAAAG0AAABmCAYAAADBPx+VAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAA3VSURBVHgB7Z27r0zdG8fX743i1bi1ikMoFMQloXRpKFFIqI7LH4BEQ+NWIkjQuSWCRIEoULk0gsK1kCBI0IhrQVT7tz/7zZo888yz1r7MnDl7z5xvsjkzs2fP3uu71nNfa7lkAsm7d++Sffv2JbNmzUqcc8m0adOSzZs3Z+/XES4ZckAWJEGWPiCxjsQNLWmQsWjRIpMseaxcuTKpG/7HP27I8P79e7dq1ars/yL4/v27S0ejqwv+cUOGEGGpKHR37tzJCEpHV9tnT58+dXXCJDdECBE2Ojrqjh071hpNECjx4cMHVycM1Uhbv359B2F79+51586daxN/+pyRkRFXKyRDAqxEp4yMlDDzXG1NPnnyJKkThoK0VFd1ELZu3TrzXKxKfW7dMBQ6bcuWLW2v0VlHjx41z717927ba22U9APcw7Nnz1oGEPeL3m3p2mTAYYnFmMOMXybPPXv2bNIPpFZr1NHn4HMw0KRBjg9NuRw95s8PEcz/6DZELQd/09C9QGq5RsmSRybqkwHGjh07OsJSsYYm3ijPpyHzoiacg35MLdDSIS/O1yM778jOTwYUkKNHWUzUWaOsylE00MyI0fcnOwIdjvtNdW/HZwNLGg+sR1kMepSNJXmIwxBZiG8tDTpEZzKg0GItNsosY8USkxDhD0Rinuiko2gfL/RbiD2LZAjU9zKQJj8RDR0vJBR1/Phx9+PHj9Z7REF4nTZkxzX4LCXHrV271qXkBAPGfP/atWvu/PnzHe4C97F48eIsRLZ9+3a3f/9+87dwP1JxaF7/3r17ba+5l4EcaVo0lj3SBq5kGTJSQmLWMjgYNei2GPT1MuMqGTDEFHzeQSP2wi/jGnkmPJ/nhccs44jvDAxpVcxnq0F6eT8h4ni/iIWpR5lPyA6ETkNXoSukvpJAD3AsXLiwpZs49+fPn5ke4j10TqYvegSfn0OnafC+Tv9ooA/JPkgQysqQNBzagXY55nO/oa1F7qvIPWkRL12WRpMWUvpVDYmxAPehxWSe8ZEXL20sadYIozfmNch4QJPAfeJgW3rNsnzphBKNJM2KKODo1rVOMRYik5ETy3ix4qWNI81qAAirizgMIc+yhTytx0JWZuNI03qsrgWlGtwjoS9XwgUhWGyhUaRZZQNNIEwCiXD16tXcAHUs79co0vSD8rrJCIW98pzvxpAWyyo3HYwqS0+H0BjStClcZJT5coMm6D2LOF8TolGJtK9fvyZpyiC5ePFi9nc/oJU4eiEP0jVoAnHa9wyJycITMP78+eMeP37sXrx44d6+fdt6f82aNdkx1pg9e3Zb5W+RSRE+n+VjksQWifvVaTKFhn5O8my63K8Qabdv33b379/PiAP//vuvW7BggZszZ072/+TJk91YgkafPn166zXB1rQHFvouAWHq9z3SEevSUerqCn2/dDCeta2jxYbr69evk4MHDyY7d+7MjhMnTiTPnz9Pfv/+nfQT2ggpO2dMF8cghuoM7Ygj5iWCqRlGFml0QC/ftGmTmzt3rmsaKDsgBSPh0/8yPeLLBihLkOKJc0jp8H8vUzcxIA1k6QJ/c78tWEyj5P3o4u9+jywNPdJi5rAH9x0KHcl4Hg570eQp3+vHXGyrmEeigzQsQsjavXt38ujRo44LQuDDhw+TW7duRS1HGgMxhNXHgflaNTOsHyKvHK5Ijo2jbFjJBQK9YwFd6RVMzfgRBmEfP37suBBm/p49e1qjEP2mwTViNRo0VJWH1deMXcNK08uUjVUu7s/zRaL+oLNxz1bpANco4npUgX4G2eFbpDFyQoQxojBCpEGSytmOH8qrH5Q9vuzD6ofQylkCUmh8DBAr+q8JCyVNtWQIidKQE9wNtLSQnS4jDSsxNHogzFuQBw4cyM61UKVsjfr3ooBkPSqqQHesUPWVtzi9/vQi1T+rJj7WiTz4Pt/l3LxUkr5P2VYZaZ4URpsE+st/dujQoaBBYokbrz/8TJNQYLSonrPS9kUaSkPeZyj1AWSj+d+VBoy1pIWVNed8P0Ll/ee5HdGRhrHhR5GGN0r4LGZBaj8oFDJitBTJzIZgFcmU0Y8ytWMZMzJOaXUSrUs5RxKnrxmbb5YXO9VGUhtpXldhEUogFr3IzIsvlpmdosVcGVGXFWp2oU9kLFL3dEkSz6NHEY1sjSRdIuDFWEhd8KxFqsRi1uM/nz9/zpxnwlESONdg6dKlbsaMGS4EHFHtjFIDHwKOo46l4TxSuxgDzi+rE2jg+BaFruOX4HXa0Nnf1lwAPufZeF8/r6zD97WK2qFnGjBxTw5qNGPxT+5T/r7/7RawFC3j4vTp09koCxkeHjqbHJqArmH5UrFKKksnxrK7FuRIs8STfBZv+luugXZ2pR/pP9Ois4z+TiMzUUkUjD0iEi1fzX8GmXyuxUBRcaUfykV0YZnlJGKQpOiGB76x5GeWkWWJc3mOrK6S7xdND+W5N6XyaRgtWJFe13GkaZnKOsYqGdOVVVbGupsyA/l7emTLHi7vwTdirNEt0qxnzAvBFcnQF16xh/TMpUuXHDowhlA9vQVraQhkudRdzOnK+04ZSP3DUhVSP61YsaLtd/ks7ZgtPcXqPqEafHkdqa84X6aCeL7YWlv6edGFHb+ZFICPlljHhg0bKuk0CSvVznWsotRu433alNdFrqG45ejoaPCaUkWERpLXjzFL2Rpllp7PJU2a/v7Ab8N05/9t27Z16KUqoFGsxnI9EosS2niSYg9SpU6B4JgTrvVW1flt1sT+0ADIJU2maXzcUTraGCRaL1Wp9rUMk16PMom8QhruxzvZIegJjFU7LLCePfS8uaQdPny4jTTL0dbee5mYokQsXTIWNY46kuMbnt8Kmec+LGWtOVIl9cT1rCB0V8WqkjAsRwta93TbwNYoGKsUSChN44lgBNCoHLHzquYKrU6qZ8lolCIN0Rh6cP0Q3U6I6IXILYOQI513hJaSKAorFpuHXJNfVlpRtmYBk1Su1obZr5dnKAO+L10Hrj3WZW+E3qh6IszE37F6EB+68mGpvKm4eb9bFrlzrok7fvr0Kfv727dvWRmdVTJHw0qiiCUSZ6wCK+7XL/AcsgNyL74DQQ730sv78Su7+t/A36MdY0sW5o40ahslXr58aZ5HtZB8GH64m9EmMZ7FpYw4T6QnrZfgenrhFxaSiSGXtPnz57e9TkNZLvTjeqhr734CNtrK41L40sUQckmj1lGKQ0rC37x544r8eNXRpnVE3ZZY7zXo8NomiO0ZUCj2uHz58rbXoZ6gc0uA+F6ZeKS/jhRDUq8MKrTho9fEkihMmhxtBI1DxKFY9XLpVcSkfoi8JGnToZO5sU5aiDQIW716ddt7ZLYtMQlhECdBGXZZMWldY5BHm5xgAroWj4C0hbYkSc/jBmggIrXJWlZM6pSETsEPGqZOndr2uuuR5rF169a2HoHPdurUKZM4CO1WTPqaDaAd+GFGKdIQkxAn9RuEWcTRyN2KSUgiSgF5aWzPTeA/lN5rZubMmR2bE4SIC4nJoltgAV/dVefZm72AtctUCJU2CMJ327hxY9t7EHbkyJFseq+EJSY16RPo3Dkq1kkr7+q0bNmyDuLQcZBEPYmHVdOBiJyIlrRDq41YPWfXOxUysi5fvtyaj+2BpcnsUV/oSoEMOk2CQGlr4ckhBwaetBhjCwH0ZHtJROPJkyc7UjcYLDjmrH7ADTEBXFfOYmB0k9oYBOjJ8b4aOYSe7QkKcYhFlq3QYLQhSidNmtS2RATwy8YOM3EQJsUjKiaWZ+vZToUQgzhkHXudb/PW5YMHD9yZM2faPsMwoc7RciYJXbGuBqJ1UIGKKLv915jsvgtJxCZDubdXr165mzdvtr1Hz5LONA8jrUwKPqsmVesKa49S3Q4WxmRPUEYdTjgiUcfUwLx589ySJUva3oMkP6IYddq6HMS4o55xBJBUeRjzfa4Zdeg56QZ43LhxoyPo7Lf1kNt7oO8wWAbNwaYjIv5lhyS7kRf96dvm5Jah8vfvX3flyhX35cuX6HfzFHOToS1H4BenCaHvO8pr8iDuwoUL7tevX+b5ZdbBair0xkFIlFDlW4ZknEClsp/TzXyAKVOmmHWFVSbDNw1l1+4f90U6IY/q4V27dpnE9bJ+v87QEydjqx/UamVVPRG+mwkNTYN+9tjkwzEx+atCm/X9WvWtDtAb68Wy9LXa1UmvCDDIpPkyOQ5ZwSzJ4jMrvFcr0rSjOUh+GcT4LSg5ugkW1Io0/SCDQBojh0hPlaJdah+tkVYrnTZowP8iq1F1TgMBBauufyB33x1v+NWFYmT5KmppgHC+NkAgbmRkpD3yn9QIseXymoTQFGQmIOKTxiZIWpvAatenVqRVXf2nTrAWMsPnKrMZHz6bJq5jvce6QK8J1cQNgKxlJapMPdZSR64/UivS9NztpkVEdKcrs5alhhWP9NeqlfWopzhZScI6QxseegZRGeg5a8C3Re1Mfl1ScP36ddcUaMuv24iOJtz7sbUjTS4qBvKmstYJoUauiuD3k5qhyr7QdUHMeCgLa1Ear9NquemdXgmum4fvJ6w1lqsuDhNrg1qSpleJK7K3TF0Q2jSd94uSZ60kK1e3qyVpQK6PVWXp2/FC3mp6jBhKKOiY2h3gtUV64TWM6wDETRPLDfSakXmH3w8g9Jlug8ZtTt4kVF0kLUYYmCCtD/DrQ5YhMGbA9L3ucdjh0y8kOHW5gU/VEEmJTcL4Pz/f7mgoAbYkAAAAAElFTkSuQmCC\"] } ] }' Response { \"model\": \"llava\", \"created_at\": \"2023-12-13T22:42:50.203334Z\", \"message\": { \"role\": \"assistant\", \"content\": \" The image features a cute, little pig with an angry facial expression. It's wearing a heart on its shirt and is waving in the air. This scene appears to be part of a drawing or sketching project.\", \"images\": null }, \"done\": true, \"total_duration\": 1668506709, \"load_duration\": 1986209, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 359682000, \"eval_count\": 83, \"eval_duration\": 1303285000 } Chat request (Reproducible outputs) Request curl http://localhost:11434/api/chat -d '{ \"model\": \"llama2\", \"messages\": [ { \"role\": \"user\", \"content\": \"Hello!\" } ], \"options\": { \"seed\": 101, \"temperature\": 0 } }' Response { \"model\": \"registry.ollama.ai/library/llama2:latest\", \"created_at\": \"2023-12-12T14:13:43.416799Z\", \"message\": { \"role\": \"assistant\", \"content\": \"Hello! How are you today?\" }, \"done\": true, \"total_duration\": 5191566416, \"load_duration\": 2154458, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 383809000, \"eval_count\": 298, \"eval_duration\": 4799921000 } Create a Model POST /api/create Create a model from a Modelfile . It is recommended to set modelfile to the content of the Modelfile rather than just set path . This is a requirement for remote create. Remote model creation must also create any file blobs, fields such as FROM and ADAPTER , explicitly with the server using Create a Blob and the value to the path indicated in the response. Parameters name : name of the model to create modelfile (optional): contents of the Modelfile stream : (optional) if false the response will be returned as a single response object, rather than a stream of objects path (optional): path to the Modelfile Examples Create a new model Create a new model from a Modelfile . Request curl http://localhost:11434/api/create -d '{ \"name\": \"mario\", \"modelfile\": \"FROM llama2\\nSYSTEM You are mario from Super Mario Bros.\" }' Response A stream of JSON objects. Notice that the final JSON object shows a \"status\": \"success\" . {\"status\":\"reading model metadata\"} {\"status\":\"creating system layer\"} {\"status\":\"using already created layer sha256:22f7f8ef5f4c791c1b03d7eb414399294764d7cc82c7e94aa81a1feb80a983a2\"} {\"status\":\"using already created layer sha256:8c17c2ebb0ea011be9981cc3922db8ca8fa61e828c5d3f44cb6ae342bf80460b\"} {\"status\":\"using already created layer sha256:7c23fb36d80141c4ab8cdbb61ee4790102ebd2bf7aeff414453177d4f2110e5d\"} {\"status\":\"using already created layer sha256:2e0493f67d0c8c9c68a8aeacdf6a38a2151cb3c4c1d42accf296e19810527988\"} {\"status\":\"using already created layer sha256:2759286baa875dc22de5394b4a925701b1896a7e3f8e53275c36f75a877a82c9\"} {\"status\":\"writing layer sha256:df30045fe90f0d750db82a058109cecd6d4de9c90a3d75b19c09e5f64580bb42\"} {\"status\":\"writing layer sha256:f18a68eb09bf925bb1b669490407c1b1251c5db98dc4d3d81f3088498ea55690\"} {\"status\":\"writing manifest\"} {\"status\":\"success\"} Check if a Blob Exists HEAD /api/blobs/:digest Ensures that the file blob used for a FROM or ADAPTER field exists on the server. This is checking your Ollama server and not Ollama.ai. Query Parameters digest : the SHA256 digest of the blob Examples Request curl -I http://localhost:11434/api/blobs/sha256:29fdb92e57cf0827ded04ae6461b5931d01fa595843f55d36f5b275a52087dd2 Response Return 200 OK if the blob exists, 404 Not Found if it does not. Create a Blob POST /api/blobs/:digest Create a blob from a file on the server. Returns the server file path. Query Parameters digest : the expected SHA256 digest of the file Examples Request curl -T model.bin -X POST http://localhost:11434/api/blobs/sha256:29fdb92e57cf0827ded04ae6461b5931d01fa595843f55d36f5b275a52087dd2 Response Return 201 Created if the blob was successfully created, 400 Bad Request if the digest used is not expected. List Local Models GET /api/tags List models that are available locally. Examples Request curl http://localhost:11434/api/tags Response A single JSON object will be returned. { \"models\": [ { \"name\": \"codellama:13b\", \"modified_at\": \"2023-11-04T14:56:49.277302595-07:00\", \"size\": 7365960935, \"digest\": \"9f438cb9cd581fc025612d27f7c1a6669ff83a8bb0ed86c94fcf4c5440555697\", \"details\": { \"format\": \"gguf\", \"family\": \"llama\", \"families\": null, \"parameter_size\": \"13B\", \"quantization_level\": \"Q4_0\" } }, { \"name\": \"llama2:latest\", \"modified_at\": \"2023-12-07T09:32:18.757212583-08:00\", \"size\": 3825819519, \"digest\": \"fe938a131f40e6f6d40083c9f0f430a515233eb2edaa6d72eb85c50d64f2300e\", \"details\": { \"format\": \"gguf\", \"family\": \"llama\", \"families\": null, \"parameter_size\": \"7B\", \"quantization_level\": \"Q4_0\" } } ] } Show Model Information POST /api/show Show information about a model including details, modelfile, template, parameters, license, and system prompt. Parameters name : name of the model to show Examples Request curl http://localhost:11434/api/show -d '{ \"name\": \"llama2\" }' Response { \"modelfile\": \"# Modelfile generated by \\\"ollama show\\\"\\n# To build a new Modelfile based on this one, replace the FROM line with:\\n# FROM llava:latest\\n\\nFROM /Users/matt/.ollama/models/blobs/sha256:200765e1283640ffbd013184bf496e261032fa75b99498a9613be4e94d63ad52\\nTEMPLATE \\\"\\\"\\\"{{ .System }}\\nUSER: {{ .Prompt }}\\nASSSISTANT: \\\"\\\"\\\"\\nPARAMETER num_ctx 4096\\nPARAMETER stop \\\"\\u003c/s\\u003e\\\"\\nPARAMETER stop \\\"USER:\\\"\\nPARAMETER stop \\\"ASSSISTANT:\\\"\", \"parameters\": \"num_ctx 4096\\nstop \\u003c/s\\u003e\\nstop USER:\\nstop ASSSISTANT:\", \"template\": \"{{ .System }}\\nUSER: {{ .Prompt }}\\nASSSISTANT: \", \"details\": { \"format\": \"gguf\", \"family\": \"llama\", \"families\": [\"llama\", \"clip\"], \"parameter_size\": \"7B\", \"quantization_level\": \"Q4_0\" } } Copy a Model POST /api/copy Copy a model. Creates a model with another name from an existing model. Examples Request curl http://localhost:11434/api/copy -d '{ \"source\": \"llama2\", \"destination\": \"llama2-backup\" }' Response Returns a 200 OK if successful, or a 404 Not Found if the source model doesn't exist. Delete a Model DELETE /api/delete Delete a model and its data. Parameters name : model name to delete Examples Request curl -X DELETE http://localhost:11434/api/delete -d '{ \"name\": \"llama2:13b\" }' Response Returns a 200 OK if successful, 404 Not Found if the model to be deleted doesn't exist. Pull a Model POST /api/pull Download a model from the ollama library. Cancelled pulls are resumed from where they left off, and multiple calls will share the same download progress. Parameters name : name of the model to pull insecure : (optional) allow insecure connections to the library. Only use this if you are pulling from your own library during development. stream : (optional) if false the response will be returned as a single response object, rather than a stream of objects Examples Request curl http://localhost:11434/api/pull -d '{ \"name\": \"llama2\" }' Response If stream is not specified, or set to true , a stream of JSON objects is returned: The first object is the manifest: { \"status\": \"pulling manifest\" } Then there is a series of downloading responses. Until any of the download is completed, the completed key may not be included. The number of files to be downloaded depends on the number of layers specified in the manifest. { \"status\": \"downloading digestname\", \"digest\": \"digestname\", \"total\": 2142590208, \"completed\": 241970 } After all the files are downloaded, the final responses are: { \"status\": \"verifying sha256 digest\" } { \"status\": \"writing manifest\" } { \"status\": \"removing any unused layers\" } { \"status\": \"success\" } if stream is set to false, then the response is a single JSON object: { \"status\": \"success\" } Push a Model POST /api/push Upload a model to a model library. Requires registering for ollama.ai and adding a public key first. Parameters name : name of the model to push in the form of /: insecure : (optional) allow insecure connections to the library. Only use this if you are pushing to your library during development. stream : (optional) if false the response will be returned as a single response object, rather than a stream of objects Examples Request curl http://localhost:11434/api/push -d '{ \"name\": \"mattw/pygmalion:latest\" }' Response If stream is not specified, or set to true , a stream of JSON objects is returned: { \"status\": \"retrieving manifest\" } and then: { \"status\": \"starting upload\", \"digest\": \"sha256:bc07c81de745696fdf5afca05e065818a8149fb0c77266fb584d9b2cba3711ab\", \"total\": 1928429856 } Then there is a series of uploading responses: { \"status\": \"starting upload\", \"digest\": \"sha256:bc07c81de745696fdf5afca05e065818a8149fb0c77266fb584d9b2cba3711ab\", \"total\": 1928429856 } Finally, when the upload is complete: {\"status\":\"pushing manifest\"} {\"status\":\"success\"} If stream is set to false , then the response is a single JSON object: { \"status\": \"success\" } Generate Embeddings POST /api/embeddings Generate embeddings from a model Parameters model : name of model to generate embeddings from prompt : text to generate embeddings for Advanced parameters: options : additional model parameters listed in the documentation for the Modelfile such as temperature keep_alive : controls how long the model will stay loaded into memory following the request (default: 5m ) Examples Request curl http://localhost:11434/api/embeddings -d '{ \"model\": \"all-minilm\", \"prompt\": \"Here is an article about llamas...\" }' Response { \"embedding\": [ 0.5670403838157654, 0.009260174818336964, 0.23178744316101074, -0.2916173040866852, -0.8924556970596313, 0.8785552978515625, -0.34576427936553955, 0.5742510557174683, -0.04222835972905159, -0.137906014919281 ] }","title":"ollama"},{"location":"ollama/#api","text":"","title":"API"},{"location":"ollama/#endpoints","text":"Generate a completion Generate a chat completion Create a Model List Local Models Show Model Information Copy a Model Delete a Model Pull a Model Push a Model Generate Embeddings","title":"Endpoints"},{"location":"ollama/#conventions","text":"","title":"Conventions"},{"location":"ollama/#model-names","text":"Model names follow a model:tag format, where model can have an optional namespace such as example/model . Some examples are orca-mini:3b-q4_1 and llama2:70b . The tag is optional and, if not provided, will default to latest . The tag is used to identify a specific version.","title":"Model names"},{"location":"ollama/#durations","text":"All durations are returned in nanoseconds.","title":"Durations"},{"location":"ollama/#streaming-responses","text":"Certain endpoints stream responses as JSON objects and can optional return non-streamed responses.","title":"Streaming responses"},{"location":"ollama/#generate-a-completion","text":"POST /api/generate Generate a response for a given prompt with a provided model. This is a streaming endpoint, so there will be a series of responses. The final response object will include statistics and additional data from the request.","title":"Generate a completion"},{"location":"ollama/#parameters","text":"model : (required) the model name prompt : the prompt to generate a response for images : (optional) a list of base64-encoded images (for multimodal models such as llava ) Advanced parameters (optional): format : the format to return a response in. Currently the only accepted value is json options : additional model parameters listed in the documentation for the Modelfile such as temperature system : system message to (overrides what is defined in the Modelfile ) template : the prompt template to use (overrides what is defined in the Modelfile ) context : the context parameter returned from a previous request to /generate , this can be used to keep a short conversational memory stream : if false the response will be returned as a single response object, rather than a stream of objects raw : if true no formatting will be applied to the prompt. You may choose to use the raw parameter if you are specifying a full templated prompt in your request to the API keep_alive : controls how long the model will stay loaded into memory following the request (default: 5m )","title":"Parameters"},{"location":"ollama/#json-mode","text":"Enable JSON mode by setting the format parameter to json . This will structure the response as a valid JSON object. See the JSON mode example below. Note: it's important to instruct the model to use JSON in the prompt . Otherwise, the model may generate large amounts whitespace.","title":"JSON mode"},{"location":"ollama/#examples","text":"","title":"Examples"},{"location":"ollama/#generate-request-streaming","text":"","title":"Generate request (Streaming)"},{"location":"ollama/#request","text":"curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\", \"prompt\": \"Why is the sky blue?\" }'","title":"Request"},{"location":"ollama/#response","text":"A stream of JSON objects is returned: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T08:52:19.385406455-07:00\", \"response\": \"The\", \"done\": false } The final response in the stream also includes additional data about the generation: total_duration : time spent generating the response load_duration : time spent in nanoseconds loading the model prompt_eval_count : number of tokens in the prompt prompt_eval_duration : time spent in nanoseconds evaluating the prompt eval_count : number of tokens the response eval_duration : time in nanoseconds spent generating the response context : an encoding of the conversation used in this response, this can be sent in the next request to keep a conversational memory response : empty if the response was streamed, if not streamed, this will contain the full response To calculate how fast the response is generated in tokens per second (token/s), divide eval_count / eval_duration . { \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"response\": \"\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 10706818083, \"load_duration\": 6338219291, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 130079000, \"eval_count\": 259, \"eval_duration\": 4232710000 }","title":"Response"},{"location":"ollama/#request-no-streaming","text":"","title":"Request (No streaming)"},{"location":"ollama/#request_1","text":"A response can be received in one reply when streaming is off. curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\", \"prompt\": \"Why is the sky blue?\", \"stream\": false }'","title":"Request"},{"location":"ollama/#response_1","text":"If stream is set to false , the response will be a single JSON object: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"response\": \"The sky is blue because it is the color of the sky.\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 5043500667, \"load_duration\": 5025959, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 325953000, \"eval_count\": 290, \"eval_duration\": 4709213000 }","title":"Response"},{"location":"ollama/#request-json-mode","text":"When format is set to json , the output will always be a well-formed JSON object. It's important to also instruct the model to respond in JSON.","title":"Request (JSON mode)"},{"location":"ollama/#request_2","text":"curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\", \"prompt\": \"What color is the sky at different times of the day? Respond using JSON\", \"format\": \"json\", \"stream\": false }'","title":"Request"},{"location":"ollama/#response_2","text":"{ \"model\": \"llama2\", \"created_at\": \"2023-11-09T21:07:55.186497Z\", \"response\": \"{\\n\\\"morning\\\": {\\n\\\"color\\\": \\\"blue\\\"\\n},\\n\\\"noon\\\": {\\n\\\"color\\\": \\\"blue-gray\\\"\\n},\\n\\\"afternoon\\\": {\\n\\\"color\\\": \\\"warm gray\\\"\\n},\\n\\\"evening\\\": {\\n\\\"color\\\": \\\"orange\\\"\\n}\\n}\\n\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 4648158584, \"load_duration\": 4071084, \"prompt_eval_count\": 36, \"prompt_eval_duration\": 439038000, \"eval_count\": 180, \"eval_duration\": 4196918000 } The value of response will be a string containing JSON similar to: { \"morning\": { \"color\": \"blue\" }, \"noon\": { \"color\": \"blue-gray\" }, \"afternoon\": { \"color\": \"warm gray\" }, \"evening\": { \"color\": \"orange\" } }","title":"Response"},{"location":"ollama/#request-with-images","text":"To submit images to multimodal models such as llava or bakllava , provide a list of base64-encoded images :","title":"Request (with images)"},{"location":"ollama/#request_3","text":"curl http://localhost:11434/api/generate -d '{ \"model\": \"llava\", \"prompt\":\"What is in this picture?\", \"stream\": false, \"images\": [\"iVBORw0KGgoAAAANSUhEUgAAAG0AAABmCAYAAADBPx+VAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAA3VSURBVHgB7Z27r0zdG8fX743i1bi1ikMoFMQloXRpKFFIqI7LH4BEQ+NWIkjQuSWCRIEoULk0gsK1kCBI0IhrQVT7tz/7zZo888yz1r7MnDl7z5xvsjkzs2fP3uu71nNfa7lkAsm7d++Sffv2JbNmzUqcc8m0adOSzZs3Z+/XES4ZckAWJEGWPiCxjsQNLWmQsWjRIpMseaxcuTKpG/7HP27I8P79e7dq1ars/yL4/v27S0ejqwv+cUOGEGGpKHR37tzJCEpHV9tnT58+dXXCJDdECBE2Ojrqjh071hpNECjx4cMHVycM1Uhbv359B2F79+51586daxN/+pyRkRFXKyRDAqxEp4yMlDDzXG1NPnnyJKkThoK0VFd1ELZu3TrzXKxKfW7dMBQ6bcuWLW2v0VlHjx41z717927ba22U9APcw7Nnz1oGEPeL3m3p2mTAYYnFmMOMXybPPXv2bNIPpFZr1NHn4HMw0KRBjg9NuRw95s8PEcz/6DZELQd/09C9QGq5RsmSRybqkwHGjh07OsJSsYYm3ijPpyHzoiacg35MLdDSIS/O1yM778jOTwYUkKNHWUzUWaOsylE00MyI0fcnOwIdjvtNdW/HZwNLGg+sR1kMepSNJXmIwxBZiG8tDTpEZzKg0GItNsosY8USkxDhD0Rinuiko2gfL/RbiD2LZAjU9zKQJj8RDR0vJBR1/Phx9+PHj9Z7REF4nTZkxzX4LCXHrV271qXkBAPGfP/atWvu/PnzHe4C97F48eIsRLZ9+3a3f/9+87dwP1JxaF7/3r17ba+5l4EcaVo0lj3SBq5kGTJSQmLWMjgYNei2GPT1MuMqGTDEFHzeQSP2wi/jGnkmPJ/nhccs44jvDAxpVcxnq0F6eT8h4ni/iIWpR5lPyA6ETkNXoSukvpJAD3AsXLiwpZs49+fPn5ke4j10TqYvegSfn0OnafC+Tv9ooA/JPkgQysqQNBzagXY55nO/oa1F7qvIPWkRL12WRpMWUvpVDYmxAPehxWSe8ZEXL20sadYIozfmNch4QJPAfeJgW3rNsnzphBKNJM2KKODo1rVOMRYik5ETy3ix4qWNI81qAAirizgMIc+yhTytx0JWZuNI03qsrgWlGtwjoS9XwgUhWGyhUaRZZQNNIEwCiXD16tXcAHUs79co0vSD8rrJCIW98pzvxpAWyyo3HYwqS0+H0BjStClcZJT5coMm6D2LOF8TolGJtK9fvyZpyiC5ePFi9nc/oJU4eiEP0jVoAnHa9wyJycITMP78+eMeP37sXrx44d6+fdt6f82aNdkx1pg9e3Zb5W+RSRE+n+VjksQWifvVaTKFhn5O8my63K8Qabdv33b379/PiAP//vuvW7BggZszZ072/+TJk91YgkafPn166zXB1rQHFvouAWHq9z3SEevSUerqCn2/dDCeta2jxYbr69evk4MHDyY7d+7MjhMnTiTPnz9Pfv/+nfQT2ggpO2dMF8cghuoM7Ygj5iWCqRlGFml0QC/ftGmTmzt3rmsaKDsgBSPh0/8yPeLLBihLkOKJc0jp8H8vUzcxIA1k6QJ/c78tWEyj5P3o4u9+jywNPdJi5rAH9x0KHcl4Hg570eQp3+vHXGyrmEeigzQsQsjavXt38ujRo44LQuDDhw+TW7duRS1HGgMxhNXHgflaNTOsHyKvHK5Ijo2jbFjJBQK9YwFd6RVMzfgRBmEfP37suBBm/p49e1qjEP2mwTViNRo0VJWH1deMXcNK08uUjVUu7s/zRaL+oLNxz1bpANco4npUgX4G2eFbpDFyQoQxojBCpEGSytmOH8qrH5Q9vuzD6ofQylkCUmh8DBAr+q8JCyVNtWQIidKQE9wNtLSQnS4jDSsxNHogzFuQBw4cyM61UKVsjfr3ooBkPSqqQHesUPWVtzi9/vQi1T+rJj7WiTz4Pt/l3LxUkr5P2VYZaZ4URpsE+st/dujQoaBBYokbrz/8TJNQYLSonrPS9kUaSkPeZyj1AWSj+d+VBoy1pIWVNed8P0Ll/ee5HdGRhrHhR5GGN0r4LGZBaj8oFDJitBTJzIZgFcmU0Y8ytWMZMzJOaXUSrUs5RxKnrxmbb5YXO9VGUhtpXldhEUogFr3IzIsvlpmdosVcGVGXFWp2oU9kLFL3dEkSz6NHEY1sjSRdIuDFWEhd8KxFqsRi1uM/nz9/zpxnwlESONdg6dKlbsaMGS4EHFHtjFIDHwKOo46l4TxSuxgDzi+rE2jg+BaFruOX4HXa0Nnf1lwAPufZeF8/r6zD97WK2qFnGjBxTw5qNGPxT+5T/r7/7RawFC3j4vTp09koCxkeHjqbHJqArmH5UrFKKksnxrK7FuRIs8STfBZv+luugXZ2pR/pP9Ois4z+TiMzUUkUjD0iEi1fzX8GmXyuxUBRcaUfykV0YZnlJGKQpOiGB76x5GeWkWWJc3mOrK6S7xdND+W5N6XyaRgtWJFe13GkaZnKOsYqGdOVVVbGupsyA/l7emTLHi7vwTdirNEt0qxnzAvBFcnQF16xh/TMpUuXHDowhlA9vQVraQhkudRdzOnK+04ZSP3DUhVSP61YsaLtd/ks7ZgtPcXqPqEafHkdqa84X6aCeL7YWlv6edGFHb+ZFICPlljHhg0bKuk0CSvVznWsotRu433alNdFrqG45ejoaPCaUkWERpLXjzFL2Rpllp7PJU2a/v7Ab8N05/9t27Z16KUqoFGsxnI9EosS2niSYg9SpU6B4JgTrvVW1flt1sT+0ADIJU2maXzcUTraGCRaL1Wp9rUMk16PMom8QhruxzvZIegJjFU7LLCePfS8uaQdPny4jTTL0dbee5mYokQsXTIWNY46kuMbnt8Kmec+LGWtOVIl9cT1rCB0V8WqkjAsRwta93TbwNYoGKsUSChN44lgBNCoHLHzquYKrU6qZ8lolCIN0Rh6cP0Q3U6I6IXILYOQI513hJaSKAorFpuHXJNfVlpRtmYBk1Su1obZr5dnKAO+L10Hrj3WZW+E3qh6IszE37F6EB+68mGpvKm4eb9bFrlzrok7fvr0Kfv727dvWRmdVTJHw0qiiCUSZ6wCK+7XL/AcsgNyL74DQQ730sv78Su7+t/A36MdY0sW5o40ahslXr58aZ5HtZB8GH64m9EmMZ7FpYw4T6QnrZfgenrhFxaSiSGXtPnz57e9TkNZLvTjeqhr734CNtrK41L40sUQckmj1lGKQ0rC37x544r8eNXRpnVE3ZZY7zXo8NomiO0ZUCj2uHz58rbXoZ6gc0uA+F6ZeKS/jhRDUq8MKrTho9fEkihMmhxtBI1DxKFY9XLpVcSkfoi8JGnToZO5sU5aiDQIW716ddt7ZLYtMQlhECdBGXZZMWldY5BHm5xgAroWj4C0hbYkSc/jBmggIrXJWlZM6pSETsEPGqZOndr2uuuR5rF169a2HoHPdurUKZM4CO1WTPqaDaAd+GFGKdIQkxAn9RuEWcTRyN2KSUgiSgF5aWzPTeA/lN5rZubMmR2bE4SIC4nJoltgAV/dVefZm72AtctUCJU2CMJ327hxY9t7EHbkyJFseq+EJSY16RPo3Dkq1kkr7+q0bNmyDuLQcZBEPYmHVdOBiJyIlrRDq41YPWfXOxUysi5fvtyaj+2BpcnsUV/oSoEMOk2CQGlr4ckhBwaetBhjCwH0ZHtJROPJkyc7UjcYLDjmrH7ADTEBXFfOYmB0k9oYBOjJ8b4aOYSe7QkKcYhFlq3QYLQhSidNmtS2RATwy8YOM3EQJsUjKiaWZ+vZToUQgzhkHXudb/PW5YMHD9yZM2faPsMwoc7RciYJXbGuBqJ1UIGKKLv915jsvgtJxCZDubdXr165mzdvtr1Hz5LONA8jrUwKPqsmVesKa49S3Q4WxmRPUEYdTjgiUcfUwLx589ySJUva3oMkP6IYddq6HMS4o55xBJBUeRjzfa4Zdeg56QZ43LhxoyPo7Lf1kNt7oO8wWAbNwaYjIv5lhyS7kRf96dvm5Jah8vfvX3flyhX35cuX6HfzFHOToS1H4BenCaHvO8pr8iDuwoUL7tevX+b5ZdbBair0xkFIlFDlW4ZknEClsp/TzXyAKVOmmHWFVSbDNw1l1+4f90U6IY/q4V27dpnE9bJ+v87QEydjqx/UamVVPRG+mwkNTYN+9tjkwzEx+atCm/X9WvWtDtAb68Wy9LXa1UmvCDDIpPkyOQ5ZwSzJ4jMrvFcr0rSjOUh+GcT4LSg5ugkW1Io0/SCDQBojh0hPlaJdah+tkVYrnTZowP8iq1F1TgMBBauufyB33x1v+NWFYmT5KmppgHC+NkAgbmRkpD3yn9QIseXymoTQFGQmIOKTxiZIWpvAatenVqRVXf2nTrAWMsPnKrMZHz6bJq5jvce6QK8J1cQNgKxlJapMPdZSR64/UivS9NztpkVEdKcrs5alhhWP9NeqlfWopzhZScI6QxseegZRGeg5a8C3Re1Mfl1ScP36ddcUaMuv24iOJtz7sbUjTS4qBvKmstYJoUauiuD3k5qhyr7QdUHMeCgLa1Ear9NquemdXgmum4fvJ6w1lqsuDhNrg1qSpleJK7K3TF0Q2jSd94uSZ60kK1e3qyVpQK6PVWXp2/FC3mp6jBhKKOiY2h3gtUV64TWM6wDETRPLDfSakXmH3w8g9Jlug8ZtTt4kVF0kLUYYmCCtD/DrQ5YhMGbA9L3ucdjh0y8kOHW5gU/VEEmJTcL4Pz/f7mgoAbYkAAAAAElFTkSuQmCC\"] }'","title":"Request"},{"location":"ollama/#response_3","text":"{ \"model\": \"llava\", \"created_at\": \"2023-11-03T15:36:02.583064Z\", \"response\": \"A happy cartoon character, which is cute and cheerful.\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 2938432250, \"load_duration\": 2559292, \"prompt_eval_count\": 1, \"prompt_eval_duration\": 2195557000, \"eval_count\": 44, \"eval_duration\": 736432000 }","title":"Response"},{"location":"ollama/#request-raw-mode","text":"In some cases, you may wish to bypass the templating system and provide a full prompt. In this case, you can use the raw parameter to disable templating. Also note that raw mode will not return a context.","title":"Request (Raw Mode)"},{"location":"ollama/#request_4","text":"curl http://localhost:11434/api/generate -d '{ \"model\": \"mistral\", \"prompt\": \"[INST] why is the sky blue? [/INST]\", \"raw\": true, \"stream\": false }'","title":"Request"},{"location":"ollama/#request-reproducible-outputs","text":"For reproducible outputs, set temperature to 0 and seed to a number:","title":"Request (Reproducible outputs)"},{"location":"ollama/#request_5","text":"curl http://localhost:11434/api/generate -d '{ \"model\": \"mistral\", \"prompt\": \"Why is the sky blue?\", \"options\": { \"seed\": 123, \"temperature\": 0 } }'","title":"Request"},{"location":"ollama/#response_4","text":"{ \"model\": \"mistral\", \"created_at\": \"2023-11-03T15:36:02.583064Z\", \"response\": \" The sky appears blue because of a phenomenon called Rayleigh scattering.\", \"done\": true, \"total_duration\": 8493852375, \"load_duration\": 6589624375, \"prompt_eval_count\": 14, \"prompt_eval_duration\": 119039000, \"eval_count\": 110, \"eval_duration\": 1779061000 }","title":"Response"},{"location":"ollama/#generate-request-with-options","text":"If you want to set custom options for the model at runtime rather than in the Modelfile, you can do so with the options parameter. This example sets every available option, but you can set any of them individually and omit the ones you do not want to override.","title":"Generate request (With options)"},{"location":"ollama/#request_6","text":"curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\", \"prompt\": \"Why is the sky blue?\", \"stream\": false, \"options\": { \"num_keep\": 5, \"seed\": 42, \"num_predict\": 100, \"top_k\": 20, \"top_p\": 0.9, \"tfs_z\": 0.5, \"typical_p\": 0.7, \"repeat_last_n\": 33, \"temperature\": 0.8, \"repeat_penalty\": 1.2, \"presence_penalty\": 1.5, \"frequency_penalty\": 1.0, \"mirostat\": 1, \"mirostat_tau\": 0.8, \"mirostat_eta\": 0.6, \"penalize_newline\": true, \"stop\": [\"\\n\", \"user:\"], \"numa\": false, \"num_ctx\": 1024, \"num_batch\": 2, \"num_gqa\": 1, \"num_gpu\": 1, \"main_gpu\": 0, \"low_vram\": false, \"f16_kv\": true, \"vocab_only\": false, \"use_mmap\": true, \"use_mlock\": false, \"rope_frequency_base\": 1.1, \"rope_frequency_scale\": 0.8, \"num_thread\": 8 } }'","title":"Request"},{"location":"ollama/#response_5","text":"{ \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"response\": \"The sky is blue because it is the color of the sky.\", \"done\": true, \"context\": [1, 2, 3], \"total_duration\": 4935886791, \"load_duration\": 534986708, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 107345000, \"eval_count\": 237, \"eval_duration\": 4289432000 }","title":"Response"},{"location":"ollama/#load-a-model","text":"If an empty prompt is provided, the model will be loaded into memory.","title":"Load a model"},{"location":"ollama/#request_7","text":"curl http://localhost:11434/api/generate -d '{ \"model\": \"llama2\" }'","title":"Request"},{"location":"ollama/#response_6","text":"A single JSON object is returned: { \"model\": \"llama2\", \"created_at\": \"2023-12-18T19:52:07.071755Z\", \"response\": \"\", \"done\": true }","title":"Response"},{"location":"ollama/#generate-a-chat-completion","text":"POST /api/chat Generate the next message in a chat with a provided model. This is a streaming endpoint, so there will be a series of responses. Streaming can be disabled using \"stream\": false . The final response object will include statistics and additional data from the request.","title":"Generate a chat completion"},{"location":"ollama/#parameters_1","text":"model : (required) the model name messages : the messages of the chat, this can be used to keep a chat memory The message object has the following fields: role : the role of the message, either system , user or assistant content : the content of the message images (optional): a list of images to include in the message (for multimodal models such as llava ) Advanced parameters (optional): format : the format to return a response in. Currently the only accepted value is json options : additional model parameters listed in the documentation for the Modelfile such as temperature template : the prompt template to use (overrides what is defined in the Modelfile ) stream : if false the response will be returned as a single response object, rather than a stream of objects keep_alive : controls how long the model will stay loaded into memory following the request (default: 5m )","title":"Parameters"},{"location":"ollama/#examples_1","text":"","title":"Examples"},{"location":"ollama/#chat-request-streaming","text":"","title":"Chat Request (Streaming)"},{"location":"ollama/#request_8","text":"Send a chat message with a streaming response. curl http://localhost:11434/api/chat -d '{ \"model\": \"llama2\", \"messages\": [ { \"role\": \"user\", \"content\": \"why is the sky blue?\" } ] }'","title":"Request"},{"location":"ollama/#response_7","text":"A stream of JSON objects is returned: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T08:52:19.385406455-07:00\", \"message\": { \"role\": \"assistant\", \"content\": \"The\", \"images\": null }, \"done\": false } Final response: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"done\": true, \"total_duration\": 4883583458, \"load_duration\": 1334875, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 342546000, \"eval_count\": 282, \"eval_duration\": 4535599000 }","title":"Response"},{"location":"ollama/#chat-request-no-streaming","text":"","title":"Chat request (No streaming)"},{"location":"ollama/#request_9","text":"curl http://localhost:11434/api/chat -d '{ \"model\": \"llama2\", \"messages\": [ { \"role\": \"user\", \"content\": \"why is the sky blue?\" } ], \"stream\": false }'","title":"Request"},{"location":"ollama/#response_8","text":"{ \"model\": \"registry.ollama.ai/library/llama2:latest\", \"created_at\": \"2023-12-12T14:13:43.416799Z\", \"message\": { \"role\": \"assistant\", \"content\": \"Hello! How are you today?\" }, \"done\": true, \"total_duration\": 5191566416, \"load_duration\": 2154458, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 383809000, \"eval_count\": 298, \"eval_duration\": 4799921000 }","title":"Response"},{"location":"ollama/#chat-request-with-history","text":"Send a chat message with a conversation history. You can use this same approach to start the conversation using multi-shot or chain-of-thought prompting.","title":"Chat request (With History)"},{"location":"ollama/#request_10","text":"curl http://localhost:11434/api/chat -d '{ \"model\": \"llama2\", \"messages\": [ { \"role\": \"user\", \"content\": \"why is the sky blue?\" }, { \"role\": \"assistant\", \"content\": \"due to rayleigh scattering.\" }, { \"role\": \"user\", \"content\": \"how is that different than mie scattering?\" } ] }'","title":"Request"},{"location":"ollama/#response_9","text":"A stream of JSON objects is returned: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T08:52:19.385406455-07:00\", \"message\": { \"role\": \"assistant\", \"content\": \"The\" }, \"done\": false } Final response: { \"model\": \"llama2\", \"created_at\": \"2023-08-04T19:22:45.499127Z\", \"done\": true, \"total_duration\": 8113331500, \"load_duration\": 6396458, \"prompt_eval_count\": 61, \"prompt_eval_duration\": 398801000, \"eval_count\": 468, \"eval_duration\": 7701267000 }","title":"Response"},{"location":"ollama/#chat-request-with-images","text":"","title":"Chat request (with images)"},{"location":"ollama/#request_11","text":"Send a chat message with a conversation history. curl http://localhost:11434/api/chat -d '{ \"model\": \"llava\", \"messages\": [ { \"role\": \"user\", \"content\": \"what is in this image?\", \"images\": [\"iVBORw0KGgoAAAANSUhEUgAAAG0AAABmCAYAAADBPx+VAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAA3VSURBVHgB7Z27r0zdG8fX743i1bi1ikMoFMQloXRpKFFIqI7LH4BEQ+NWIkjQuSWCRIEoULk0gsK1kCBI0IhrQVT7tz/7zZo888yz1r7MnDl7z5xvsjkzs2fP3uu71nNfa7lkAsm7d++Sffv2JbNmzUqcc8m0adOSzZs3Z+/XES4ZckAWJEGWPiCxjsQNLWmQsWjRIpMseaxcuTKpG/7HP27I8P79e7dq1ars/yL4/v27S0ejqwv+cUOGEGGpKHR37tzJCEpHV9tnT58+dXXCJDdECBE2Ojrqjh071hpNECjx4cMHVycM1Uhbv359B2F79+51586daxN/+pyRkRFXKyRDAqxEp4yMlDDzXG1NPnnyJKkThoK0VFd1ELZu3TrzXKxKfW7dMBQ6bcuWLW2v0VlHjx41z717927ba22U9APcw7Nnz1oGEPeL3m3p2mTAYYnFmMOMXybPPXv2bNIPpFZr1NHn4HMw0KRBjg9NuRw95s8PEcz/6DZELQd/09C9QGq5RsmSRybqkwHGjh07OsJSsYYm3ijPpyHzoiacg35MLdDSIS/O1yM778jOTwYUkKNHWUzUWaOsylE00MyI0fcnOwIdjvtNdW/HZwNLGg+sR1kMepSNJXmIwxBZiG8tDTpEZzKg0GItNsosY8USkxDhD0Rinuiko2gfL/RbiD2LZAjU9zKQJj8RDR0vJBR1/Phx9+PHj9Z7REF4nTZkxzX4LCXHrV271qXkBAPGfP/atWvu/PnzHe4C97F48eIsRLZ9+3a3f/9+87dwP1JxaF7/3r17ba+5l4EcaVo0lj3SBq5kGTJSQmLWMjgYNei2GPT1MuMqGTDEFHzeQSP2wi/jGnkmPJ/nhccs44jvDAxpVcxnq0F6eT8h4ni/iIWpR5lPyA6ETkNXoSukvpJAD3AsXLiwpZs49+fPn5ke4j10TqYvegSfn0OnafC+Tv9ooA/JPkgQysqQNBzagXY55nO/oa1F7qvIPWkRL12WRpMWUvpVDYmxAPehxWSe8ZEXL20sadYIozfmNch4QJPAfeJgW3rNsnzphBKNJM2KKODo1rVOMRYik5ETy3ix4qWNI81qAAirizgMIc+yhTytx0JWZuNI03qsrgWlGtwjoS9XwgUhWGyhUaRZZQNNIEwCiXD16tXcAHUs79co0vSD8rrJCIW98pzvxpAWyyo3HYwqS0+H0BjStClcZJT5coMm6D2LOF8TolGJtK9fvyZpyiC5ePFi9nc/oJU4eiEP0jVoAnHa9wyJycITMP78+eMeP37sXrx44d6+fdt6f82aNdkx1pg9e3Zb5W+RSRE+n+VjksQWifvVaTKFhn5O8my63K8Qabdv33b379/PiAP//vuvW7BggZszZ072/+TJk91YgkafPn166zXB1rQHFvouAWHq9z3SEevSUerqCn2/dDCeta2jxYbr69evk4MHDyY7d+7MjhMnTiTPnz9Pfv/+nfQT2ggpO2dMF8cghuoM7Ygj5iWCqRlGFml0QC/ftGmTmzt3rmsaKDsgBSPh0/8yPeLLBihLkOKJc0jp8H8vUzcxIA1k6QJ/c78tWEyj5P3o4u9+jywNPdJi5rAH9x0KHcl4Hg570eQp3+vHXGyrmEeigzQsQsjavXt38ujRo44LQuDDhw+TW7duRS1HGgMxhNXHgflaNTOsHyKvHK5Ijo2jbFjJBQK9YwFd6RVMzfgRBmEfP37suBBm/p49e1qjEP2mwTViNRo0VJWH1deMXcNK08uUjVUu7s/zRaL+oLNxz1bpANco4npUgX4G2eFbpDFyQoQxojBCpEGSytmOH8qrH5Q9vuzD6ofQylkCUmh8DBAr+q8JCyVNtWQIidKQE9wNtLSQnS4jDSsxNHogzFuQBw4cyM61UKVsjfr3ooBkPSqqQHesUPWVtzi9/vQi1T+rJj7WiTz4Pt/l3LxUkr5P2VYZaZ4URpsE+st/dujQoaBBYokbrz/8TJNQYLSonrPS9kUaSkPeZyj1AWSj+d+VBoy1pIWVNed8P0Ll/ee5HdGRhrHhR5GGN0r4LGZBaj8oFDJitBTJzIZgFcmU0Y8ytWMZMzJOaXUSrUs5RxKnrxmbb5YXO9VGUhtpXldhEUogFr3IzIsvlpmdosVcGVGXFWp2oU9kLFL3dEkSz6NHEY1sjSRdIuDFWEhd8KxFqsRi1uM/nz9/zpxnwlESONdg6dKlbsaMGS4EHFHtjFIDHwKOo46l4TxSuxgDzi+rE2jg+BaFruOX4HXa0Nnf1lwAPufZeF8/r6zD97WK2qFnGjBxTw5qNGPxT+5T/r7/7RawFC3j4vTp09koCxkeHjqbHJqArmH5UrFKKksnxrK7FuRIs8STfBZv+luugXZ2pR/pP9Ois4z+TiMzUUkUjD0iEi1fzX8GmXyuxUBRcaUfykV0YZnlJGKQpOiGB76x5GeWkWWJc3mOrK6S7xdND+W5N6XyaRgtWJFe13GkaZnKOsYqGdOVVVbGupsyA/l7emTLHi7vwTdirNEt0qxnzAvBFcnQF16xh/TMpUuXHDowhlA9vQVraQhkudRdzOnK+04ZSP3DUhVSP61YsaLtd/ks7ZgtPcXqPqEafHkdqa84X6aCeL7YWlv6edGFHb+ZFICPlljHhg0bKuk0CSvVznWsotRu433alNdFrqG45ejoaPCaUkWERpLXjzFL2Rpllp7PJU2a/v7Ab8N05/9t27Z16KUqoFGsxnI9EosS2niSYg9SpU6B4JgTrvVW1flt1sT+0ADIJU2maXzcUTraGCRaL1Wp9rUMk16PMom8QhruxzvZIegJjFU7LLCePfS8uaQdPny4jTTL0dbee5mYokQsXTIWNY46kuMbnt8Kmec+LGWtOVIl9cT1rCB0V8WqkjAsRwta93TbwNYoGKsUSChN44lgBNCoHLHzquYKrU6qZ8lolCIN0Rh6cP0Q3U6I6IXILYOQI513hJaSKAorFpuHXJNfVlpRtmYBk1Su1obZr5dnKAO+L10Hrj3WZW+E3qh6IszE37F6EB+68mGpvKm4eb9bFrlzrok7fvr0Kfv727dvWRmdVTJHw0qiiCUSZ6wCK+7XL/AcsgNyL74DQQ730sv78Su7+t/A36MdY0sW5o40ahslXr58aZ5HtZB8GH64m9EmMZ7FpYw4T6QnrZfgenrhFxaSiSGXtPnz57e9TkNZLvTjeqhr734CNtrK41L40sUQckmj1lGKQ0rC37x544r8eNXRpnVE3ZZY7zXo8NomiO0ZUCj2uHz58rbXoZ6gc0uA+F6ZeKS/jhRDUq8MKrTho9fEkihMmhxtBI1DxKFY9XLpVcSkfoi8JGnToZO5sU5aiDQIW716ddt7ZLYtMQlhECdBGXZZMWldY5BHm5xgAroWj4C0hbYkSc/jBmggIrXJWlZM6pSETsEPGqZOndr2uuuR5rF169a2HoHPdurUKZM4CO1WTPqaDaAd+GFGKdIQkxAn9RuEWcTRyN2KSUgiSgF5aWzPTeA/lN5rZubMmR2bE4SIC4nJoltgAV/dVefZm72AtctUCJU2CMJ327hxY9t7EHbkyJFseq+EJSY16RPo3Dkq1kkr7+q0bNmyDuLQcZBEPYmHVdOBiJyIlrRDq41YPWfXOxUysi5fvtyaj+2BpcnsUV/oSoEMOk2CQGlr4ckhBwaetBhjCwH0ZHtJROPJkyc7UjcYLDjmrH7ADTEBXFfOYmB0k9oYBOjJ8b4aOYSe7QkKcYhFlq3QYLQhSidNmtS2RATwy8YOM3EQJsUjKiaWZ+vZToUQgzhkHXudb/PW5YMHD9yZM2faPsMwoc7RciYJXbGuBqJ1UIGKKLv915jsvgtJxCZDubdXr165mzdvtr1Hz5LONA8jrUwKPqsmVesKa49S3Q4WxmRPUEYdTjgiUcfUwLx589ySJUva3oMkP6IYddq6HMS4o55xBJBUeRjzfa4Zdeg56QZ43LhxoyPo7Lf1kNt7oO8wWAbNwaYjIv5lhyS7kRf96dvm5Jah8vfvX3flyhX35cuX6HfzFHOToS1H4BenCaHvO8pr8iDuwoUL7tevX+b5ZdbBair0xkFIlFDlW4ZknEClsp/TzXyAKVOmmHWFVSbDNw1l1+4f90U6IY/q4V27dpnE9bJ+v87QEydjqx/UamVVPRG+mwkNTYN+9tjkwzEx+atCm/X9WvWtDtAb68Wy9LXa1UmvCDDIpPkyOQ5ZwSzJ4jMrvFcr0rSjOUh+GcT4LSg5ugkW1Io0/SCDQBojh0hPlaJdah+tkVYrnTZowP8iq1F1TgMBBauufyB33x1v+NWFYmT5KmppgHC+NkAgbmRkpD3yn9QIseXymoTQFGQmIOKTxiZIWpvAatenVqRVXf2nTrAWMsPnKrMZHz6bJq5jvce6QK8J1cQNgKxlJapMPdZSR64/UivS9NztpkVEdKcrs5alhhWP9NeqlfWopzhZScI6QxseegZRGeg5a8C3Re1Mfl1ScP36ddcUaMuv24iOJtz7sbUjTS4qBvKmstYJoUauiuD3k5qhyr7QdUHMeCgLa1Ear9NquemdXgmum4fvJ6w1lqsuDhNrg1qSpleJK7K3TF0Q2jSd94uSZ60kK1e3qyVpQK6PVWXp2/FC3mp6jBhKKOiY2h3gtUV64TWM6wDETRPLDfSakXmH3w8g9Jlug8ZtTt4kVF0kLUYYmCCtD/DrQ5YhMGbA9L3ucdjh0y8kOHW5gU/VEEmJTcL4Pz/f7mgoAbYkAAAAAElFTkSuQmCC\"] } ] }'","title":"Request"},{"location":"ollama/#response_10","text":"{ \"model\": \"llava\", \"created_at\": \"2023-12-13T22:42:50.203334Z\", \"message\": { \"role\": \"assistant\", \"content\": \" The image features a cute, little pig with an angry facial expression. It's wearing a heart on its shirt and is waving in the air. This scene appears to be part of a drawing or sketching project.\", \"images\": null }, \"done\": true, \"total_duration\": 1668506709, \"load_duration\": 1986209, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 359682000, \"eval_count\": 83, \"eval_duration\": 1303285000 }","title":"Response"},{"location":"ollama/#chat-request-reproducible-outputs","text":"","title":"Chat request (Reproducible outputs)"},{"location":"ollama/#request_12","text":"curl http://localhost:11434/api/chat -d '{ \"model\": \"llama2\", \"messages\": [ { \"role\": \"user\", \"content\": \"Hello!\" } ], \"options\": { \"seed\": 101, \"temperature\": 0 } }'","title":"Request"},{"location":"ollama/#response_11","text":"{ \"model\": \"registry.ollama.ai/library/llama2:latest\", \"created_at\": \"2023-12-12T14:13:43.416799Z\", \"message\": { \"role\": \"assistant\", \"content\": \"Hello! How are you today?\" }, \"done\": true, \"total_duration\": 5191566416, \"load_duration\": 2154458, \"prompt_eval_count\": 26, \"prompt_eval_duration\": 383809000, \"eval_count\": 298, \"eval_duration\": 4799921000 }","title":"Response"},{"location":"ollama/#create-a-model","text":"POST /api/create Create a model from a Modelfile . It is recommended to set modelfile to the content of the Modelfile rather than just set path . This is a requirement for remote create. Remote model creation must also create any file blobs, fields such as FROM and ADAPTER , explicitly with the server using Create a Blob and the value to the path indicated in the response.","title":"Create a Model"},{"location":"ollama/#parameters_2","text":"name : name of the model to create modelfile (optional): contents of the Modelfile stream : (optional) if false the response will be returned as a single response object, rather than a stream of objects path (optional): path to the Modelfile","title":"Parameters"},{"location":"ollama/#examples_2","text":"","title":"Examples"},{"location":"ollama/#create-a-new-model","text":"Create a new model from a Modelfile .","title":"Create a new model"},{"location":"ollama/#request_13","text":"curl http://localhost:11434/api/create -d '{ \"name\": \"mario\", \"modelfile\": \"FROM llama2\\nSYSTEM You are mario from Super Mario Bros.\" }'","title":"Request"},{"location":"ollama/#response_12","text":"A stream of JSON objects. Notice that the final JSON object shows a \"status\": \"success\" . {\"status\":\"reading model metadata\"} {\"status\":\"creating system layer\"} {\"status\":\"using already created layer sha256:22f7f8ef5f4c791c1b03d7eb414399294764d7cc82c7e94aa81a1feb80a983a2\"} {\"status\":\"using already created layer sha256:8c17c2ebb0ea011be9981cc3922db8ca8fa61e828c5d3f44cb6ae342bf80460b\"} {\"status\":\"using already created layer sha256:7c23fb36d80141c4ab8cdbb61ee4790102ebd2bf7aeff414453177d4f2110e5d\"} {\"status\":\"using already created layer sha256:2e0493f67d0c8c9c68a8aeacdf6a38a2151cb3c4c1d42accf296e19810527988\"} {\"status\":\"using already created layer sha256:2759286baa875dc22de5394b4a925701b1896a7e3f8e53275c36f75a877a82c9\"} {\"status\":\"writing layer sha256:df30045fe90f0d750db82a058109cecd6d4de9c90a3d75b19c09e5f64580bb42\"} {\"status\":\"writing layer sha256:f18a68eb09bf925bb1b669490407c1b1251c5db98dc4d3d81f3088498ea55690\"} {\"status\":\"writing manifest\"} {\"status\":\"success\"}","title":"Response"},{"location":"ollama/#check-if-a-blob-exists","text":"HEAD /api/blobs/:digest Ensures that the file blob used for a FROM or ADAPTER field exists on the server. This is checking your Ollama server and not Ollama.ai.","title":"Check if a Blob Exists"},{"location":"ollama/#query-parameters","text":"digest : the SHA256 digest of the blob","title":"Query Parameters"},{"location":"ollama/#examples_3","text":"","title":"Examples"},{"location":"ollama/#request_14","text":"curl -I http://localhost:11434/api/blobs/sha256:29fdb92e57cf0827ded04ae6461b5931d01fa595843f55d36f5b275a52087dd2","title":"Request"},{"location":"ollama/#response_13","text":"Return 200 OK if the blob exists, 404 Not Found if it does not.","title":"Response"},{"location":"ollama/#create-a-blob","text":"POST /api/blobs/:digest Create a blob from a file on the server. Returns the server file path.","title":"Create a Blob"},{"location":"ollama/#query-parameters_1","text":"digest : the expected SHA256 digest of the file","title":"Query Parameters"},{"location":"ollama/#examples_4","text":"","title":"Examples"},{"location":"ollama/#request_15","text":"curl -T model.bin -X POST http://localhost:11434/api/blobs/sha256:29fdb92e57cf0827ded04ae6461b5931d01fa595843f55d36f5b275a52087dd2","title":"Request"},{"location":"ollama/#response_14","text":"Return 201 Created if the blob was successfully created, 400 Bad Request if the digest used is not expected.","title":"Response"},{"location":"ollama/#list-local-models","text":"GET /api/tags List models that are available locally.","title":"List Local Models"},{"location":"ollama/#examples_5","text":"","title":"Examples"},{"location":"ollama/#request_16","text":"curl http://localhost:11434/api/tags","title":"Request"},{"location":"ollama/#response_15","text":"A single JSON object will be returned. { \"models\": [ { \"name\": \"codellama:13b\", \"modified_at\": \"2023-11-04T14:56:49.277302595-07:00\", \"size\": 7365960935, \"digest\": \"9f438cb9cd581fc025612d27f7c1a6669ff83a8bb0ed86c94fcf4c5440555697\", \"details\": { \"format\": \"gguf\", \"family\": \"llama\", \"families\": null, \"parameter_size\": \"13B\", \"quantization_level\": \"Q4_0\" } }, { \"name\": \"llama2:latest\", \"modified_at\": \"2023-12-07T09:32:18.757212583-08:00\", \"size\": 3825819519, \"digest\": \"fe938a131f40e6f6d40083c9f0f430a515233eb2edaa6d72eb85c50d64f2300e\", \"details\": { \"format\": \"gguf\", \"family\": \"llama\", \"families\": null, \"parameter_size\": \"7B\", \"quantization_level\": \"Q4_0\" } } ] }","title":"Response"},{"location":"ollama/#show-model-information","text":"POST /api/show Show information about a model including details, modelfile, template, parameters, license, and system prompt.","title":"Show Model Information"},{"location":"ollama/#parameters_3","text":"name : name of the model to show","title":"Parameters"},{"location":"ollama/#examples_6","text":"","title":"Examples"},{"location":"ollama/#request_17","text":"curl http://localhost:11434/api/show -d '{ \"name\": \"llama2\" }'","title":"Request"},{"location":"ollama/#response_16","text":"{ \"modelfile\": \"# Modelfile generated by \\\"ollama show\\\"\\n# To build a new Modelfile based on this one, replace the FROM line with:\\n# FROM llava:latest\\n\\nFROM /Users/matt/.ollama/models/blobs/sha256:200765e1283640ffbd013184bf496e261032fa75b99498a9613be4e94d63ad52\\nTEMPLATE \\\"\\\"\\\"{{ .System }}\\nUSER: {{ .Prompt }}\\nASSSISTANT: \\\"\\\"\\\"\\nPARAMETER num_ctx 4096\\nPARAMETER stop \\\"\\u003c/s\\u003e\\\"\\nPARAMETER stop \\\"USER:\\\"\\nPARAMETER stop \\\"ASSSISTANT:\\\"\", \"parameters\": \"num_ctx 4096\\nstop \\u003c/s\\u003e\\nstop USER:\\nstop ASSSISTANT:\", \"template\": \"{{ .System }}\\nUSER: {{ .Prompt }}\\nASSSISTANT: \", \"details\": { \"format\": \"gguf\", \"family\": \"llama\", \"families\": [\"llama\", \"clip\"], \"parameter_size\": \"7B\", \"quantization_level\": \"Q4_0\" } }","title":"Response"},{"location":"ollama/#copy-a-model","text":"POST /api/copy Copy a model. Creates a model with another name from an existing model.","title":"Copy a Model"},{"location":"ollama/#examples_7","text":"","title":"Examples"},{"location":"ollama/#request_18","text":"curl http://localhost:11434/api/copy -d '{ \"source\": \"llama2\", \"destination\": \"llama2-backup\" }'","title":"Request"},{"location":"ollama/#response_17","text":"Returns a 200 OK if successful, or a 404 Not Found if the source model doesn't exist.","title":"Response"},{"location":"ollama/#delete-a-model","text":"DELETE /api/delete Delete a model and its data.","title":"Delete a Model"},{"location":"ollama/#parameters_4","text":"name : model name to delete","title":"Parameters"},{"location":"ollama/#examples_8","text":"","title":"Examples"},{"location":"ollama/#request_19","text":"curl -X DELETE http://localhost:11434/api/delete -d '{ \"name\": \"llama2:13b\" }'","title":"Request"},{"location":"ollama/#response_18","text":"Returns a 200 OK if successful, 404 Not Found if the model to be deleted doesn't exist.","title":"Response"},{"location":"ollama/#pull-a-model","text":"POST /api/pull Download a model from the ollama library. Cancelled pulls are resumed from where they left off, and multiple calls will share the same download progress.","title":"Pull a Model"},{"location":"ollama/#parameters_5","text":"name : name of the model to pull insecure : (optional) allow insecure connections to the library. Only use this if you are pulling from your own library during development. stream : (optional) if false the response will be returned as a single response object, rather than a stream of objects","title":"Parameters"},{"location":"ollama/#examples_9","text":"","title":"Examples"},{"location":"ollama/#request_20","text":"curl http://localhost:11434/api/pull -d '{ \"name\": \"llama2\" }'","title":"Request"},{"location":"ollama/#response_19","text":"If stream is not specified, or set to true , a stream of JSON objects is returned: The first object is the manifest: { \"status\": \"pulling manifest\" } Then there is a series of downloading responses. Until any of the download is completed, the completed key may not be included. The number of files to be downloaded depends on the number of layers specified in the manifest. { \"status\": \"downloading digestname\", \"digest\": \"digestname\", \"total\": 2142590208, \"completed\": 241970 } After all the files are downloaded, the final responses are: { \"status\": \"verifying sha256 digest\" } { \"status\": \"writing manifest\" } { \"status\": \"removing any unused layers\" } { \"status\": \"success\" } if stream is set to false, then the response is a single JSON object: { \"status\": \"success\" }","title":"Response"},{"location":"ollama/#push-a-model","text":"POST /api/push Upload a model to a model library. Requires registering for ollama.ai and adding a public key first.","title":"Push a Model"},{"location":"ollama/#parameters_6","text":"name : name of the model to push in the form of /: insecure : (optional) allow insecure connections to the library. Only use this if you are pushing to your library during development. stream : (optional) if false the response will be returned as a single response object, rather than a stream of objects","title":"Parameters"},{"location":"ollama/#examples_10","text":"","title":"Examples"},{"location":"ollama/#request_21","text":"curl http://localhost:11434/api/push -d '{ \"name\": \"mattw/pygmalion:latest\" }'","title":"Request"},{"location":"ollama/#response_20","text":"If stream is not specified, or set to true , a stream of JSON objects is returned: { \"status\": \"retrieving manifest\" } and then: { \"status\": \"starting upload\", \"digest\": \"sha256:bc07c81de745696fdf5afca05e065818a8149fb0c77266fb584d9b2cba3711ab\", \"total\": 1928429856 } Then there is a series of uploading responses: { \"status\": \"starting upload\", \"digest\": \"sha256:bc07c81de745696fdf5afca05e065818a8149fb0c77266fb584d9b2cba3711ab\", \"total\": 1928429856 } Finally, when the upload is complete: {\"status\":\"pushing manifest\"} {\"status\":\"success\"} If stream is set to false , then the response is a single JSON object: { \"status\": \"success\" }","title":"Response"},{"location":"ollama/#generate-embeddings","text":"POST /api/embeddings Generate embeddings from a model","title":"Generate Embeddings"},{"location":"ollama/#parameters_7","text":"model : name of model to generate embeddings from prompt : text to generate embeddings for Advanced parameters: options : additional model parameters listed in the documentation for the Modelfile such as temperature keep_alive : controls how long the model will stay loaded into memory following the request (default: 5m )","title":"Parameters"},{"location":"ollama/#examples_11","text":"","title":"Examples"},{"location":"ollama/#request_22","text":"curl http://localhost:11434/api/embeddings -d '{ \"model\": \"all-minilm\", \"prompt\": \"Here is an article about llamas...\" }'","title":"Request"},{"location":"ollama/#response_21","text":"{ \"embedding\": [ 0.5670403838157654, 0.009260174818336964, 0.23178744316101074, -0.2916173040866852, -0.8924556970596313, 0.8785552978515625, -0.34576427936553955, 0.5742510557174683, -0.04222835972905159, -0.137906014919281 ] }","title":"Response"},{"location":"poetry/","text":"poetry # Install poetry curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python - # Install dependencies poetry install # Run poetry run python main.py # Run tests poetry run pytest # Run tests with coverage and open in browser poetry run pytest --cov=src --cov-report=html && open htmlcov/index.html # Fix poetry shell not activating virtualenv source \"$( poetry env list --full-path | grep Activated | cut -d' ' -f1 )/bin/activate\" # Add from requirements.txt poetry add $(cat requirements.txt) # or cat requirements.txt | xargs poetry add","title":"poetry"},{"location":"poetry/#poetry","text":"# Install poetry curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python - # Install dependencies poetry install # Run poetry run python main.py # Run tests poetry run pytest # Run tests with coverage and open in browser poetry run pytest --cov=src --cov-report=html && open htmlcov/index.html # Fix poetry shell not activating virtualenv source \"$( poetry env list --full-path | grep Activated | cut -d' ' -f1 )/bin/activate\" # Add from requirements.txt poetry add $(cat requirements.txt) # or cat requirements.txt | xargs poetry add","title":"poetry"},{"location":"rag/","text":"RAG Huggingface Retriever used to get documents from vector queries. It retrieves the documents embeddings as well as the documents contents, and it formats them to be used with a RagModel. Examples: # To load the default \"wiki_dpr\" dataset with 21M passages from wikipedia (index name is 'compressed' or 'exact') from transformers import RagRetriever retriever = RagRetriever.from_pretrained( \"facebook/dpr-ctx_encoder-single-nq-base\", dataset=\"wiki_dpr\", index_name=\"compressed\" ) # To load your own indexed dataset built with the datasets library. More info on how to build the indexed dataset in examples/rag/use_own_knowledge_dataset.py from transformers import RagRetriever dataset = ( ... ) # dataset must be a datasets.Datasets object with columns \"title\", \"text\" and \"embeddings\", and it must have a faiss index retriever = RagRetriever.from_pretrained(\"facebook/dpr-ctx_encoder-single-nq-base\", indexed_dataset=dataset) # To load your own indexed dataset built with the datasets library that was saved on disk. More info in examples/rag/use_own_knowledge_dataset.py from transformers import RagRetriever dataset_path = \"path/to/my/dataset\" # dataset saved via *dataset.save_to_disk(...)* index_path = \"path/to/my/index.faiss\" # faiss index saved via *dataset.get_index(\"embeddings\").save(...)* retriever = RagRetriever.from_pretrained( \"facebook/dpr-ctx_encoder-single-nq-base\", index_name=\"custom\", passages_path=dataset_path, index_path=index_path, ) # To load the legacy index built originally for Rag's paper from transformers import RagRetriever retriever = RagRetriever.from_pretrained(\"facebook/dpr-ctx_encoder-single-nq-base\", index_name=\"legacy\") RAG is a seq2seq model which encapsulates two core components: a question encoder and a generator. During a forward pass, we encode the input with the question encoder and pass it to the retriever to extract relevant context documents. The documents are then prepended to the input. Such contextualized inputs is passed to the generator. from transformers import AutoTokenizer, RagRetriever, RagModel import torch tokenizer = AutoTokenizer.from_pretrained(\"facebook/rag-token-base\") retriever = RagRetriever.from_pretrained( \"facebook/rag-token-base\", index_name=\"exact\", use_dummy_dataset=True ) # initialize with RagRetriever to do everything in one forward call model = RagModel.from_pretrained(\"facebook/rag-token-base\", retriever=retriever) inputs = tokenizer(\"How many people live in Paris?\", return_tensors=\"pt\") outputs = model(input_ids=inputs[\"input_ids\"]) from transformers import AutoTokenizer, RagRetriever, RagSequenceForGeneration import torch tokenizer = AutoTokenizer.from_pretrained(\"facebook/rag-sequence-nq\") retriever = RagRetriever.from_pretrained( \"facebook/rag-sequence-nq\", index_name=\"exact\", use_dummy_dataset=True ) # initialize with RagRetriever to do everything in one forward call model = RagSequenceForGeneration.from_pretrained(\"facebook/rag-token-nq\", retriever=retriever) inputs = tokenizer(\"How many people live in Paris?\", return_tensors=\"pt\") targets = tokenizer(text_target=\"In Paris, there are 10 million people.\", return_tensors=\"pt\") input_ids = inputs[\"input_ids\"] labels = targets[\"input_ids\"] outputs = model(input_ids=input_ids, labels=labels) # or use retriever separately model = RagSequenceForGeneration.from_pretrained(\"facebook/rag-sequence-nq\", use_dummy_dataset=True) # 1. Encode question_hidden_states = model.question_encoder(input_ids)[0] # 2. Retrieve docs_dict = retriever(input_ids.numpy(), question_hidden_states.detach().numpy(), return_tensors=\"pt\") doc_scores = torch.bmm( question_hidden_states.unsqueeze(1), docs_dict[\"retrieved_doc_embeds\"].float().transpose(1, 2) ).squeeze(1) # 3. Forward to generator outputs = model( context_input_ids=docs_dict[\"context_input_ids\"], context_attention_mask=docs_dict[\"context_attention_mask\"], doc_scores=doc_scores, decoder_input_ids=labels, ) langchain https://python.langchain.com/docs/expression_language/cookbook/retrieval Let's look at adding in a retrieval step to a prompt and LLM, which adds up to a \"retrieval-augmented generation\" chain pip install langchain openai faiss-cpu tiktoken `````` ```python from operator import itemgetter from langchain.prompts import ChatPromptTemplate from langchain.chat_models import ChatOpenAI from langchain.embeddings import OpenAIEmbeddings from langchain.schema.output_parser import StrOutputParser from langchain.schema.runnable import RunnablePassthrough from langchain.vectorstores import FAISS vectorstore = FAISS.from_texts([\"harrison worked at kensho\"], embedding=OpenAIEmbeddings()) retriever = vectorstore.as_retriever() template = \"\"\"Answer the question based only on the following context: {context} Question: {question} \"\"\" prompt = ChatPromptTemplate.from_template(template) model = ChatOpenAI() chain = ( {\"context\": retriever, \"question\": RunnablePassthrough()} | prompt | model | StrOutputParser() ) chain.invoke(\"where did harrison work?\") template = \"\"\"Answer the question based only on the following context: {context} Question: {question} Answer in the following language: {language} \"\"\" prompt = ChatPromptTemplate.from_template(template) chain = { \"context\": itemgetter(\"question\") | retriever, \"question\": itemgetter(\"question\"), \"language\": itemgetter(\"language\") } | prompt | model | StrOutputParser() chain.invoke({\"question\": \"where did harrison work\", \"language\": \"italian\"})","title":"rag"},{"location":"rag/#rag","text":"","title":"RAG"},{"location":"rag/#huggingface","text":"Retriever used to get documents from vector queries. It retrieves the documents embeddings as well as the documents contents, and it formats them to be used with a RagModel. Examples: # To load the default \"wiki_dpr\" dataset with 21M passages from wikipedia (index name is 'compressed' or 'exact') from transformers import RagRetriever retriever = RagRetriever.from_pretrained( \"facebook/dpr-ctx_encoder-single-nq-base\", dataset=\"wiki_dpr\", index_name=\"compressed\" ) # To load your own indexed dataset built with the datasets library. More info on how to build the indexed dataset in examples/rag/use_own_knowledge_dataset.py from transformers import RagRetriever dataset = ( ... ) # dataset must be a datasets.Datasets object with columns \"title\", \"text\" and \"embeddings\", and it must have a faiss index retriever = RagRetriever.from_pretrained(\"facebook/dpr-ctx_encoder-single-nq-base\", indexed_dataset=dataset) # To load your own indexed dataset built with the datasets library that was saved on disk. More info in examples/rag/use_own_knowledge_dataset.py from transformers import RagRetriever dataset_path = \"path/to/my/dataset\" # dataset saved via *dataset.save_to_disk(...)* index_path = \"path/to/my/index.faiss\" # faiss index saved via *dataset.get_index(\"embeddings\").save(...)* retriever = RagRetriever.from_pretrained( \"facebook/dpr-ctx_encoder-single-nq-base\", index_name=\"custom\", passages_path=dataset_path, index_path=index_path, ) # To load the legacy index built originally for Rag's paper from transformers import RagRetriever retriever = RagRetriever.from_pretrained(\"facebook/dpr-ctx_encoder-single-nq-base\", index_name=\"legacy\") RAG is a seq2seq model which encapsulates two core components: a question encoder and a generator. During a forward pass, we encode the input with the question encoder and pass it to the retriever to extract relevant context documents. The documents are then prepended to the input. Such contextualized inputs is passed to the generator. from transformers import AutoTokenizer, RagRetriever, RagModel import torch tokenizer = AutoTokenizer.from_pretrained(\"facebook/rag-token-base\") retriever = RagRetriever.from_pretrained( \"facebook/rag-token-base\", index_name=\"exact\", use_dummy_dataset=True ) # initialize with RagRetriever to do everything in one forward call model = RagModel.from_pretrained(\"facebook/rag-token-base\", retriever=retriever) inputs = tokenizer(\"How many people live in Paris?\", return_tensors=\"pt\") outputs = model(input_ids=inputs[\"input_ids\"]) from transformers import AutoTokenizer, RagRetriever, RagSequenceForGeneration import torch tokenizer = AutoTokenizer.from_pretrained(\"facebook/rag-sequence-nq\") retriever = RagRetriever.from_pretrained( \"facebook/rag-sequence-nq\", index_name=\"exact\", use_dummy_dataset=True ) # initialize with RagRetriever to do everything in one forward call model = RagSequenceForGeneration.from_pretrained(\"facebook/rag-token-nq\", retriever=retriever) inputs = tokenizer(\"How many people live in Paris?\", return_tensors=\"pt\") targets = tokenizer(text_target=\"In Paris, there are 10 million people.\", return_tensors=\"pt\") input_ids = inputs[\"input_ids\"] labels = targets[\"input_ids\"] outputs = model(input_ids=input_ids, labels=labels) # or use retriever separately model = RagSequenceForGeneration.from_pretrained(\"facebook/rag-sequence-nq\", use_dummy_dataset=True) # 1. Encode question_hidden_states = model.question_encoder(input_ids)[0] # 2. Retrieve docs_dict = retriever(input_ids.numpy(), question_hidden_states.detach().numpy(), return_tensors=\"pt\") doc_scores = torch.bmm( question_hidden_states.unsqueeze(1), docs_dict[\"retrieved_doc_embeds\"].float().transpose(1, 2) ).squeeze(1) # 3. Forward to generator outputs = model( context_input_ids=docs_dict[\"context_input_ids\"], context_attention_mask=docs_dict[\"context_attention_mask\"], doc_scores=doc_scores, decoder_input_ids=labels, )","title":"Huggingface"},{"location":"rag/#langchain","text":"https://python.langchain.com/docs/expression_language/cookbook/retrieval Let's look at adding in a retrieval step to a prompt and LLM, which adds up to a \"retrieval-augmented generation\" chain pip install langchain openai faiss-cpu tiktoken `````` ```python from operator import itemgetter from langchain.prompts import ChatPromptTemplate from langchain.chat_models import ChatOpenAI from langchain.embeddings import OpenAIEmbeddings from langchain.schema.output_parser import StrOutputParser from langchain.schema.runnable import RunnablePassthrough from langchain.vectorstores import FAISS vectorstore = FAISS.from_texts([\"harrison worked at kensho\"], embedding=OpenAIEmbeddings()) retriever = vectorstore.as_retriever() template = \"\"\"Answer the question based only on the following context: {context} Question: {question} \"\"\" prompt = ChatPromptTemplate.from_template(template) model = ChatOpenAI() chain = ( {\"context\": retriever, \"question\": RunnablePassthrough()} | prompt | model | StrOutputParser() ) chain.invoke(\"where did harrison work?\") template = \"\"\"Answer the question based only on the following context: {context} Question: {question} Answer in the following language: {language} \"\"\" prompt = ChatPromptTemplate.from_template(template) chain = { \"context\": itemgetter(\"question\") | retriever, \"question\": itemgetter(\"question\"), \"language\": itemgetter(\"language\") } | prompt | model | StrOutputParser() chain.invoke({\"question\": \"where did harrison work\", \"language\": \"italian\"})","title":"langchain"},{"location":"random/","text":"Random Quartet Hobby became dream. People who cry while eating always survive. https://github.com/aisingapore/PeekingDuck Problem: peeking duck is a opensource modular framework, built for Computer Vision inference. A need for training custom computer vision models with custom datasets. Task: To create a configurable training pipeline for Pytorch and Tensorflow framework for custom datasets for various tasks like classification and object detection. Actions: We used the C4 diagram to blueprint the architecture and selected Hydra as the best configuration framework. The training pipeline was built using the SOLID principles for sustainable development. Results: A highly configurable training pipeline for both Tensorflow and pytorch. Peeking Duck is an open-source modular framework built for Computer Vision inference. The team created a configurable training pipeline for Pytorch and Tensorflow frameworks for custom datasets for various tasks like classification and object detection. They used the C4 diagram to blueprint the architecture and selected Hydra as the best configuration framework. The training pipeline was built using the SOLID principles for sustainable development. The result is a highly configurable training pipeline for both Tensorflow and Pytorch. SOLID is an acronym for five design principles intended to make software designs more understandable, flexible and maintainable. The principles are Single Responsibility Principle (SRP), Open-Closed Principle (OCP), Liskov Substitution Principle (LSP), Interface Segregation Principle (ISP), and Dependency Inversion Principle (DIP).123 SRP: A class should have only one reason to change. OCP: Software entities should be open for extension but closed for modification. LSP: Objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program. ISP: A client should never be forced to implement an interface that it doesn\u2019t use or clients shouldn\u2019t be forced to depend on methods they do not use. DIP: High-level modules should not depend on low-level modules. Both should depend on abstractions. Abstractions should not depend on details. Details should depend on abstractions.","title":"random"},{"location":"random/#random","text":"","title":"Random"},{"location":"random/#quartet","text":"Hobby became dream. People who cry while eating always survive. https://github.com/aisingapore/PeekingDuck Problem: peeking duck is a opensource modular framework, built for Computer Vision inference. A need for training custom computer vision models with custom datasets. Task: To create a configurable training pipeline for Pytorch and Tensorflow framework for custom datasets for various tasks like classification and object detection. Actions: We used the C4 diagram to blueprint the architecture and selected Hydra as the best configuration framework. The training pipeline was built using the SOLID principles for sustainable development. Results: A highly configurable training pipeline for both Tensorflow and pytorch. Peeking Duck is an open-source modular framework built for Computer Vision inference. The team created a configurable training pipeline for Pytorch and Tensorflow frameworks for custom datasets for various tasks like classification and object detection. They used the C4 diagram to blueprint the architecture and selected Hydra as the best configuration framework. The training pipeline was built using the SOLID principles for sustainable development. The result is a highly configurable training pipeline for both Tensorflow and Pytorch. SOLID is an acronym for five design principles intended to make software designs more understandable, flexible and maintainable. The principles are Single Responsibility Principle (SRP), Open-Closed Principle (OCP), Liskov Substitution Principle (LSP), Interface Segregation Principle (ISP), and Dependency Inversion Principle (DIP).123 SRP: A class should have only one reason to change. OCP: Software entities should be open for extension but closed for modification. LSP: Objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program. ISP: A client should never be forced to implement an interface that it doesn\u2019t use or clients shouldn\u2019t be forced to depend on methods they do not use. DIP: High-level modules should not depend on low-level modules. Both should depend on abstractions. Abstractions should not depend on details. Details should depend on abstractions.","title":"Quartet"},{"location":"resume/","text":"Resume \u2584\u2584\u258c \u2590 \u2584\u258c \u2590 \u2584 \u2584\u2584 \u2022 \u2584\u2584\u2584\u2584\u2584\u2584\u2584\u2584 . \u2584\u2584\u00b7 \u2584 \u2022\u2584 \u2022 \u258c \u2584 \u00b7. \u2584\u2584\u2584 . \u2590 \u2584 \u2584\u2584 \u2022 \u2588\u2588\u00b7 \u2588\u258c\u2590\u2588\u25aa \u2022\u2588\u258c\u2590\u2588\u2590\u2588 \u2580 \u25aa \u2022\u2588\u2588 \u2580\u2584.\u2580\u00b7\u2590\u2588 \u258c\u25aa\u2588\u258c\u2584\u258c\u25aa \u00b7\u2588\u2588 \u2590\u2588\u2588\u2588\u25aa\u2580\u2584.\u2580\u00b7\u2022\u2588\u258c\u2590\u2588\u2590\u2588 \u2580 \u25aa \u2588\u2588\u25aa\u2590\u2588\u2590\u2590\u258c \u2584\u2588\u2580\u2584 \u2590\u2588\u2590\u2590\u258c\u2584\u2588 \u2580\u2588\u2584 \u2590\u2588.\u25aa\u2590\u2580\u2580\u25aa\u2584\u2588\u2588 \u2584\u2584\u2590\u2580\u2580\u2584\u00b7 \u2590\u2588 \u258c\u2590\u258c\u2590\u2588\u00b7\u2590\u2580\u2580\u25aa\u2584\u2590\u2588\u2590\u2590\u258c\u2584\u2588 \u2580\u2588\u2584 \u2590\u2588\u258c\u2588\u2588\u2590\u2588\u258c\u2590\u2588\u258c.\u2590\u258c\u2588\u2588\u2590\u2588\u258c\u2590\u2588\u2584\u25aa\u2590\u2588 \u2590\u2588\u258c\u00b7\u2590\u2588\u2584\u2584\u258c\u2590\u2588\u2588\u2588\u258c\u2590\u2588.\u2588\u258c \u2588\u2588 \u2588\u2588\u258c\u2590\u2588\u258c\u2590\u2588\u2584\u2584\u258c\u2588\u2588\u2590\u2588\u258c\u2590\u2588\u2584\u25aa\u2590\u2588 \u2580\u2580\u2580\u2580 \u2580\u25aa \u2580\u2588\u2584\u2580\u25aa\u2580\u2580 \u2588\u25aa\u00b7\u2580\u2580\u2580\u2580 \u2580\u2580\u2580 \u2580\u2580\u2580 \u00b7\u2580\u2580\u2580 \u00b7\u2580 \u2580 \u2580\u2580 \u2588\u25aa\u2580\u2580\u2580 \u2580\u2580\u2580 \u2580\u2580 \u2588\u25aa\u00b7\u2580\u2580\u2580\u2580 20 years of software engineering experience. Reverse engineering codes with strong learning attitude. Particulars Full Name as per passport: Wong Teck Meng Nationality: Singaporean \"Email: \" gfuryhawk@gmail.com Code repositories https://github.com/furyhawk https://www.kaggle.com/teckmengwong Experiences Employment history : 20+ years Software Engineering Period Employer Positions Responsibilities 2023-2024 UParcel AI Engineer Software Engineering 2022-2023 AI Singapore Associate AI Engineer MLOps 2020-2021 NCS Fiori developer SAP Edge Implementation 2019-2020 BHP Developer Fiori Project Implementation 2006-2019 NCS Senior Consultant SAP Consultant/ABAP/Fiori 2000-2006 Creative Associate Engineer Software QA UParcel : 2023 - present AI Engineer Optimize the delivery route for UParcel using large data analysis and parameter fine tuning of heuristic algorithm. Achieved 20% increase in driver acceptance rate of listed jobs. Roll out optimized last mile delivery to Malaysia market. Project management of CD/CI pipeline for UParcel using AWS techstack CDK . AI Singapore : 2022 - 2023 : 1 yrs Associate AI Engineer Scoping requirements for content generation of event planning for a government agency using Langchain with prompt engineering . Implement MLOps using Pytorch and Tensorflow framework. Easy configurable( Hydra ) ML pipeline for tasks like Image Classification and Object Detection using SOLID engineering principles. https://github.com/aisingapore/PeekingDuck Research and design industrial defect detection POC for American multinational company using OpenVINO . Deploying to Nvidia Jetson Xavier NX Developer kit. Mentor batch 12 and 13 on Computer Vision . AI Apprentice Out of hundreds applicants, one of the selected 19 in batch 11 . 2 months of deepskilling in AI Engineering and 7 months of on-the-job training on a real-world AI problem from the industry. Deploy scalable distributed container platforms using Kubernetes in GCP . Career transition : 2021 - 2022 : 1 yrs Self-hosted applications and api services on [https://furyhawk.lol/] using Docker Swarm on 3 Raspberry Pi. Transitioning hosted applications and api services to Kubernetes on using Talos and Proxmox . Team Fight Tactics Strategy Application github.com/furyhawk/tftchamp Dockerized application to show gamers meta of current patch using feature importances. Frontend implemented using React and Zustand . NCS : 2020 - 2021 : 1 yrs Fiori developer Implement SAP Edge applications using Agile methodology. Maintenance Dashboard for Singapore Air force F16/Apache Squadron on Edge devices to improve turnaround reliability. BHP : 2019 - 2020 : 1 yr 2 mos Developer Global implementation of Resource Scheduler application that allows for mass scheduling of work to personnel, taking into account work restrictions and capacity availability. Decrease Resource Scheduler Time-on-task , User Error Rate and bolstered safety compliance. NCS : 2008 - 2019 : 10 yrs 7 mos Senior Consultant Successfully implemented 11 cycles of end to end Enterprise Resource Planning ( SAP ) projects, total worth over 100 millions dollars in wide range of business domains(Material Planning, Procurement, Logistics, Finance and Control, Human Resource and Plant Maintenance). Bolstered productivity and add value after each project cycles using Software Engineering best practice . Expertise in implementing projects using Frontend ( Fiori ) and Backend ( ABAP ) frameworks. Spearheaded Agile Scrum methodology to project team members. Restored critical production database outage under time pressure applying root cause analysis techniques . Creative : 2001 - 2008 : 7 yrs 1 mos Associate Engineer Software/Hardware Quality Assurance for world first mp3 devices. Education Qualifications Period Discipline/University 2011-2017 Bachelor of Science IT and Business (ERP) Singapore University of Social Sciences Professional Qualification Certificate Verification \u2013 AI Professionals Association (AIP) Udacity Nanodegree Program - Machine Learning Engineer SMU Certificate in Artificial Intelligence Module 1 - Building AI Capability with Basic Coding for Business SAP Certified Application Associate \u2013 Procurement with SAP ERP 6.0 EHP6 C_FIORDEV_20 Development Associate \u2013 SAP Fiori Application Developer Hobbies - Open source projects","title":"Resume"},{"location":"resume/#resume","text":"\u2584\u2584\u258c \u2590 \u2584\u258c \u2590 \u2584 \u2584\u2584 \u2022 \u2584\u2584\u2584\u2584\u2584\u2584\u2584\u2584 . \u2584\u2584\u00b7 \u2584 \u2022\u2584 \u2022 \u258c \u2584 \u00b7. \u2584\u2584\u2584 . \u2590 \u2584 \u2584\u2584 \u2022 \u2588\u2588\u00b7 \u2588\u258c\u2590\u2588\u25aa \u2022\u2588\u258c\u2590\u2588\u2590\u2588 \u2580 \u25aa \u2022\u2588\u2588 \u2580\u2584.\u2580\u00b7\u2590\u2588 \u258c\u25aa\u2588\u258c\u2584\u258c\u25aa \u00b7\u2588\u2588 \u2590\u2588\u2588\u2588\u25aa\u2580\u2584.\u2580\u00b7\u2022\u2588\u258c\u2590\u2588\u2590\u2588 \u2580 \u25aa \u2588\u2588\u25aa\u2590\u2588\u2590\u2590\u258c \u2584\u2588\u2580\u2584 \u2590\u2588\u2590\u2590\u258c\u2584\u2588 \u2580\u2588\u2584 \u2590\u2588.\u25aa\u2590\u2580\u2580\u25aa\u2584\u2588\u2588 \u2584\u2584\u2590\u2580\u2580\u2584\u00b7 \u2590\u2588 \u258c\u2590\u258c\u2590\u2588\u00b7\u2590\u2580\u2580\u25aa\u2584\u2590\u2588\u2590\u2590\u258c\u2584\u2588 \u2580\u2588\u2584 \u2590\u2588\u258c\u2588\u2588\u2590\u2588\u258c\u2590\u2588\u258c.\u2590\u258c\u2588\u2588\u2590\u2588\u258c\u2590\u2588\u2584\u25aa\u2590\u2588 \u2590\u2588\u258c\u00b7\u2590\u2588\u2584\u2584\u258c\u2590\u2588\u2588\u2588\u258c\u2590\u2588.\u2588\u258c \u2588\u2588 \u2588\u2588\u258c\u2590\u2588\u258c\u2590\u2588\u2584\u2584\u258c\u2588\u2588\u2590\u2588\u258c\u2590\u2588\u2584\u25aa\u2590\u2588 \u2580\u2580\u2580\u2580 \u2580\u25aa \u2580\u2588\u2584\u2580\u25aa\u2580\u2580 \u2588\u25aa\u00b7\u2580\u2580\u2580\u2580 \u2580\u2580\u2580 \u2580\u2580\u2580 \u00b7\u2580\u2580\u2580 \u00b7\u2580 \u2580 \u2580\u2580 \u2588\u25aa\u2580\u2580\u2580 \u2580\u2580\u2580 \u2580\u2580 \u2588\u25aa\u00b7\u2580\u2580\u2580\u2580 20 years of software engineering experience. Reverse engineering codes with strong learning attitude.","title":"Resume"},{"location":"resume/#particulars","text":"Full Name as per passport: Wong Teck Meng Nationality: Singaporean \"Email: \" gfuryhawk@gmail.com Code repositories https://github.com/furyhawk https://www.kaggle.com/teckmengwong","title":"Particulars"},{"location":"resume/#experiences","text":"","title":"Experiences"},{"location":"resume/#employment-history-20-years-software-engineering","text":"Period Employer Positions Responsibilities 2023-2024 UParcel AI Engineer Software Engineering 2022-2023 AI Singapore Associate AI Engineer MLOps 2020-2021 NCS Fiori developer SAP Edge Implementation 2019-2020 BHP Developer Fiori Project Implementation 2006-2019 NCS Senior Consultant SAP Consultant/ABAP/Fiori 2000-2006 Creative Associate Engineer Software QA","title":"Employment history : 20+ years Software Engineering"},{"location":"resume/#uparcel-2023-present","text":"AI Engineer Optimize the delivery route for UParcel using large data analysis and parameter fine tuning of heuristic algorithm. Achieved 20% increase in driver acceptance rate of listed jobs. Roll out optimized last mile delivery to Malaysia market. Project management of CD/CI pipeline for UParcel using AWS techstack CDK .","title":"UParcel : 2023 - present"},{"location":"resume/#ai-singapore-2022-2023-1-yrs","text":"Associate AI Engineer Scoping requirements for content generation of event planning for a government agency using Langchain with prompt engineering . Implement MLOps using Pytorch and Tensorflow framework. Easy configurable( Hydra ) ML pipeline for tasks like Image Classification and Object Detection using SOLID engineering principles. https://github.com/aisingapore/PeekingDuck Research and design industrial defect detection POC for American multinational company using OpenVINO . Deploying to Nvidia Jetson Xavier NX Developer kit. Mentor batch 12 and 13 on Computer Vision . AI Apprentice Out of hundreds applicants, one of the selected 19 in batch 11 . 2 months of deepskilling in AI Engineering and 7 months of on-the-job training on a real-world AI problem from the industry. Deploy scalable distributed container platforms using Kubernetes in GCP .","title":"AI Singapore : 2022 - 2023 : 1 yrs"},{"location":"resume/#career-transition-2021-2022-1-yrs","text":"Self-hosted applications and api services on [https://furyhawk.lol/] using Docker Swarm on 3 Raspberry Pi. Transitioning hosted applications and api services to Kubernetes on using Talos and Proxmox . Team Fight Tactics Strategy Application github.com/furyhawk/tftchamp Dockerized application to show gamers meta of current patch using feature importances. Frontend implemented using React and Zustand .","title":"Career transition : 2021 - 2022 : 1 yrs"},{"location":"resume/#ncs-2020-2021-1-yrs","text":"Fiori developer Implement SAP Edge applications using Agile methodology. Maintenance Dashboard for Singapore Air force F16/Apache Squadron on Edge devices to improve turnaround reliability.","title":"NCS : 2020 - 2021 : 1 yrs"},{"location":"resume/#bhp-2019-2020-1-yr-2-mos","text":"Developer Global implementation of Resource Scheduler application that allows for mass scheduling of work to personnel, taking into account work restrictions and capacity availability. Decrease Resource Scheduler Time-on-task , User Error Rate and bolstered safety compliance.","title":"BHP : 2019 - 2020 : 1 yr 2 mos"},{"location":"resume/#ncs-2008-2019-10-yrs-7-mos","text":"Senior Consultant Successfully implemented 11 cycles of end to end Enterprise Resource Planning ( SAP ) projects, total worth over 100 millions dollars in wide range of business domains(Material Planning, Procurement, Logistics, Finance and Control, Human Resource and Plant Maintenance). Bolstered productivity and add value after each project cycles using Software Engineering best practice . Expertise in implementing projects using Frontend ( Fiori ) and Backend ( ABAP ) frameworks. Spearheaded Agile Scrum methodology to project team members. Restored critical production database outage under time pressure applying root cause analysis techniques .","title":"NCS : 2008 - 2019 : 10 yrs 7 mos"},{"location":"resume/#creative-2001-2008-7-yrs-1-mos","text":"Associate Engineer Software/Hardware Quality Assurance for world first mp3 devices.","title":"Creative : 2001 - 2008 : 7 yrs 1 mos"},{"location":"resume/#education-qualifications","text":"Period Discipline/University 2011-2017 Bachelor of Science IT and Business (ERP) Singapore University of Social Sciences","title":"Education Qualifications"},{"location":"resume/#professional-qualification","text":"Certificate Verification \u2013 AI Professionals Association (AIP) Udacity Nanodegree Program - Machine Learning Engineer SMU Certificate in Artificial Intelligence Module 1 - Building AI Capability with Basic Coding for Business SAP Certified Application Associate \u2013 Procurement with SAP ERP 6.0 EHP6 C_FIORDEV_20 Development Associate \u2013 SAP Fiori Application Developer","title":"Professional Qualification"},{"location":"resume/#hobbies","text":"- Open source projects","title":"Hobbies"},{"location":"scrape_pad/","text":"majaro \u201croot=PARTUUID=1d7d1003-8f39-41da-9db4-f46a07e06335 rw rootwait audit=0 splash plymouth.ignore-serial-consoles\u201d sway \u201croot=PARTUUID=e9389d07-71a8-44a0-b383-1fbe0c22001a\" \"rw rootwait audit=0 splash plymouth.ignore-serial-consoles\u201d Full Name as per passport: Wong Teck Meng Nationality: Singaporean \"Email: \" gfuryhawk@gmail.com Yolox python tools/train.py -f exps/example/custom/nano.py -d 1 -b 2 -c yolox_nano.pth targets.shape torch.Size([2, 120, 5]) inps.shape torch.Size([2, 3, 416, 416]) question org, dept, position level. what are the expectation of me after 6 months. when do I expectate a reply. weak point briefly. stage of job interview before preparation during connection how are you? Thanks for you for asking, I have being looking forward to meeting you. Smile before exit. tell me about yourself 2-3 stories relevant to jobs. 90-120s. STARS. aspiration. overqualifed immediately fulfill the role, take more roles. ideal job work life balance yrself in 2 yrs growing to fill in others roles. (never about title, promotion) I will really appreciate it if you make the first offer. after follow up the next day on thank you note. misc cp /etc/i3status.conf ~/.config/i3status/config systemctl --user start docker-desktop sudo groupadd docker sudo usermod -aG docker $USER","title":"scrape_pad"},{"location":"scrape_pad/#majaro","text":"\u201croot=PARTUUID=1d7d1003-8f39-41da-9db4-f46a07e06335 rw rootwait audit=0 splash plymouth.ignore-serial-consoles\u201d","title":"majaro"},{"location":"scrape_pad/#sway","text":"\u201croot=PARTUUID=e9389d07-71a8-44a0-b383-1fbe0c22001a\" \"rw rootwait audit=0 splash plymouth.ignore-serial-consoles\u201d Full Name as per passport: Wong Teck Meng Nationality: Singaporean \"Email: \" gfuryhawk@gmail.com","title":"sway"},{"location":"scrape_pad/#yolox","text":"python tools/train.py -f exps/example/custom/nano.py -d 1 -b 2 -c yolox_nano.pth targets.shape torch.Size([2, 120, 5]) inps.shape torch.Size([2, 3, 416, 416])","title":"Yolox"},{"location":"scrape_pad/#question","text":"org, dept, position level. what are the expectation of me after 6 months. when do I expectate a reply. weak point briefly.","title":"question"},{"location":"scrape_pad/#stage-of-job-interview","text":"","title":"stage of job interview"},{"location":"scrape_pad/#before","text":"preparation","title":"before"},{"location":"scrape_pad/#during","text":"connection how are you? Thanks for you for asking, I have being looking forward to meeting you. Smile before exit.","title":"during"},{"location":"scrape_pad/#tell-me-about-yourself","text":"2-3 stories relevant to jobs. 90-120s. STARS. aspiration.","title":"tell me about yourself"},{"location":"scrape_pad/#overqualifed","text":"immediately fulfill the role, take more roles.","title":"overqualifed"},{"location":"scrape_pad/#ideal-job","text":"work life balance","title":"ideal job"},{"location":"scrape_pad/#yrself-in-2-yrs","text":"growing to fill in others roles. (never about title, promotion) I will really appreciate it if you make the first offer.","title":"yrself in 2 yrs"},{"location":"scrape_pad/#after","text":"follow up the next day on thank you note.","title":"after"},{"location":"scrape_pad/#misc","text":"cp /etc/i3status.conf ~/.config/i3status/config systemctl --user start docker-desktop sudo groupadd docker sudo usermod -aG docker $USER","title":"misc"},{"location":"smb/","text":"Samba Samba is the standard Windows interoperability suite of programs for Linux and Unix. Since 1992, Samba has provided secure, stable and fast file and print services for all clients using the SMB/CIFS protocol, such as all versions of DOS and Windows, OS/2, Linux and many others. To share files through Samba, see #Server section; to access files shared through Samba on other machines, please see #Client section. Server cd /mnt/ sudo mkdir media nano /home/user/.credentials sudo chmod 400 /home/user/.credentials sudo mount -t cifs -o rw,vers=3.0,credentials=/home/user/.credentials //coco.local/media/ /mnt/media sudo nano /etc/fstab # smbclient -L hostname -U% # //coco.local/media /mnt/media cifs vers=3.0,credentials=/home/user/.credentials Installation Install the package. Samba is configured in the configuration file, which is extensively documented in . Because the package does not provide this file, one needs to create it before starting . A documented example as in from the Samba git repository may be used to setup . Enabling and starting services To provide basic file sharing through SMB, enable/start . See for details. If you want to make your server accessible via NetBIOS host name, set the desired name in the option in and enable/start . See for details. Make the server discoverable Install the package, then enable/start to make the samba server discoverable with Zeroconf . It should work for most non-Windows file managers (macOS Finder, various GUI-based file managers on Linux & BSD etc.) If is not running, the server will still be accessible, just not discoverable, i.e. it will not show up in file managers, but you can still connect to the server directly by IP or domain. Windows Explorer relies on the WS-Discovery protocol instead; see #Windows 1709 or up does not discover the samba server in Network view . Configure firewall If you are using a firewall , do not forget to open required ports (usually 137-139 + 445). For a complete list, see Samba port usage . UFW Rule A Ufw App Profile for SMB/CIFS is included by default with the default installation of UFW in . Allow Samba by running as root. If you deleted the profile, create/edit and add the following content: [Samba] title=LanManager-like file and printer server for Unix description=The Samba software suite is a collection of programs that implements the SMB/CIFS protocol for unix systems, allowing you to serve files and printers to Windows, NT, OS/2 and DOS clients. This protocol is sometimes also referred to as the LanManager or NetBIOS protocol. ports=137,138/udp|139,445/tcp Then load the profile into UFW run as root. Then finally, allow Samba by running as root. firewalld service To configure firewalld to allow Samba in the home zone, run: # firewall-cmd --permanent --add-service={samba,samba-client,samba-dc} --zone=home The three services listed are: : for sharing files with others. : to browse shares on other machines on the network. : for Samba/Active Directory domain controller . ensures the changes remain after is restarted . Basic configuration User management The following section describes creating a local (tdbsam) database of Samba users. For user authentication and other purposes, Samba can also be bound to an Active Directory domain, can itself serve as an Active Directory domain controller, or can be used with an LDAP server. Adding a user Samba requires a Linux user account - you may use an existing user account or create a new one . Although the user name is shared with Linux system, Samba uses a password separate from that of the Linux user accounts. Replace with the chosen Samba user account: # smbpasswd -a samba_user Depending on the server role , existing File permissions and attributes may need to be altered for the Samba user account. If you want the new user only to be allowed to remotely access the file server shares through Samba, you can restrict other login options\uff1a disabling shell - disabling SSH logons - edit , change option Also see Security for hardening your system. Listing users Samba users can be listed using the command: # pdbedit -L -v Changing user password To change a user password, use : # smbpasswd samba_user Creating an anonymous share 1. Create a Linux user which anonymous Samba users will be mapped to. # useradd guest -s /bin/nologin 2. Add the following to : Anonymous users will now be mapped to the Linux user and have the ability to access any directories defined in , which is configured to be in the example above. Make sure that the Linux user has the proper permissions to access files in . Also, make sure shares have been properly defined as per the Share Definitions section of smb.conf.default . Advanced configuration Enable symlink following Then, restart . Enable server-side copy for macOS clients Server-side copy eliminates the need to transfer data between the server and the client when copying files on the server. This is enabled by default, but it doesn't work with macOS clients. If you have macOS clients, you need to add the following configuration to and then restart . Enable Usershares Usershares is a feature that gives non-root users the capability to add, modify, and delete their own share definitions. See . Create a directory for usershares: Create a user group : Change the owner of the directory to and the group to : Change the permissions of the directory so that users in the group can create files. This command also sets sticky bit , which is important to prevent users from deleting usershares of other users: Set the following parameters in the configuration file: Add the user to the sambashare group. Replace with the name of your user: # gpasswd sambashare -a your_username Restart and services. Log out and log back in. If you want to share paths inside your home directory you must make it accessible for the group others . In the GUI, you can use Thunar or Dolphin - right click on any directory and share it on the network. In the CLI, use one of the following commands, replacing italic sharename , user , ... : # net usershare add sharename abspath [ comment ] [ user :{R|D|F}] [guest_ok={y|n}] # net usershare delete sharename # net usershare list wildcard-sharename # net usershare info wildcard-sharename Set and forcing permissions Permissions may be applied to both the server and shares: See for a full overview of possible permission flags and settings. Restrict protocols for better security Append and in to force usage of a minimum and maximum protocol: See in for an overview of supported protocols. For compatibility with older clients and/or servers, you might need to set or , but please note that this makes you vulnerable to exploits in SMB1 including ransomware attacks. Clients using may need to specify the correct , e.g.: # mount -t cifs // SERVER / sharename /mnt/ mountpoint -o username= username ,password= password ,iocharset= utf8 ,vers= 3.1.1 See for more information. Use native SMB transport encryption Native SMB transport encryption is available in SMB version 3.0 or newer. Clients supporting this type of encryption include Windows 8 and newer, Windows server 2012 and newer, and smbclient of Samba 4.1 and newer. To use native SMB transport encryption by default, set the parameter globally and/or by share. Possible values are , (default value), , or : To configure encryption for on the client side, use the option . See for more information, especially the paragraphs Effects for SMB1 and Effects for SMB2 . Disable printer sharing By default Samba shares printers configured using CUPS . If you do not want printers to be shared, use the following settings: Block certain file extensions on Samba share Samba offers an option to block files with certain patterns, like file extensions. This option can be used to prevent dissemination of viruses or to dissuade users from wasting space with certain files. More information about this option can be found in . Improve throughput The default settings should be sufficient for most users. However setting the 'socket options' correct can improve performance, but getting them wrong can degrade it by just as much. Test the effect before making any large changes. Read the man page before applying any of the options listed below. The following settings should be appended to the section of . Setting a deadtime is useful to stop a server's resources from being exhausted by a large number of inactive connections: deadtime = 30 The usage of sendfile may make more efficient use of the system CPU's and cause Samba to be faster: use sendfile = yes Setting min receivefile size allows zero-copy writes directly from network socket buffers into the filesystem buffer cache (if available). It may improve performance but user testing is recommended: min receivefile size = 16384 Increasing the receive/send buffers size and socket optimize flags might be useful to improve throughput. It is recommended to test each flag separately as it may cause issues on some networks: socket options = IPTOS_LOWDELAY TCP_NODELAY IPTOS_THROUGHPUT SO_RCVBUF=131072 SO_SNDBUF=131072 Enable access for old clients/devices Latest versions of Samba no longer offer older authentication methods and protocols which are still used by some older clients (IP cameras, etc). These devices usually require Samba server to allow NTMLv1 authentication and NT1 version of the protocol, known as CIFS. For these devices to work with latest Samba, you need to add these two configuration parameters into section: server min protocol = NT1 ntlm auth = yes Anonymous/guest access to a share requires just the first parameter. If the old device will access with username and password, you also need the add the second line too. Enable Spotlight searching Spotlight allows supporting clients (e.g. MacOS Finder) to quickly search shared files. Install and start/enable OpenSearch . Install , configure the directories you want to index in , and start/enable for periodic indexing. Edit as described in the Samba wiki to enable Spotlight per share, and restart to apply the changes. Client Install for an -like command line interface. See for commonly used commands. For a lightweight alternative (without support for listing public shares, etc.), install that provides . Depending on the desktop environment , GUI methods may be available. See #File manager configuration for use with a file manager. List public shares The following command lists public shares on a server: $ smbclient -L hostname -U% Alternatively, running will show a tree diagram of all the shares. It uses broadcast queries and is therefore not advisable on a network with a lot of computers, but can be helpful for diagnosing if you have the correct sharename. The () option suppresses the password prompt. NetBIOS/WINS host names Samba clients handle NetBIOS host names automatically by default (the behavior is controlled by the option in ). Other programs (including ) typically use Name Service Switch , which does not handle NetBIOS by default. The package provides a libnss driver to resolve NetBIOS host names. To use it, install it along with the package (which provides the winbindd daemon), start/enable and add to the line in : Now, during host resolving (e.g. when using or just ), winbindd will resolve the host name by sending queries using NetBIOS Name Service (NBNS, also known as WINS) protocol. By default it sends a broadcast query to your local network. If you have a WINS server, you can add to and restart , then winbindd and other Samba clients will send unicast queries to the specified IP. If you want to resolve your local host name (specified in the option in ), start/enable , which will handle incoming queries. You can test WINS resolution with . By default it sends broadcast queries to your local network regardless of the option. Note that WINS resolution requires incoming traffic originating from port 137. Disable NetBIOS/WINS support When not using NetBIOS/WINS host name resolution, it may be preferred to disable this protocol: /etc/samba/smb.conf [global] disable netbios = yes dns proxy = no Finally disable / stop . Manual mounting Mount the share using as . Not all the options listed below are needed or desirable: # mount --mkdir -t cifs // SERVER / sharename /mnt/ mountpoint -o username= username ,password= password ,workgroup= workgroup ,iocharset= utf8 ,uid= username ,gid= group The options uid and gid corresponds to the local (e.g. client) user / user group to have read/write access on the given path. Note : If the uid and gid being used does not match the user of the server, the forceuid and forcegid options may be helpful. However note permissions assigned to a file when forceuid or forcegid are in effect may not reflect the the real (server) permissions. See the File And Directory Ownership And Permissions section in mount.cifs(8) \u00a7\u202fFILE AND DIRECTORY OWNERSHIP AND PERMISSIONS for more information. To mount a Windows share without authentication, use \"username=*\". Warning : Using uid and/or gid as mount options may cause I/O errors, it is recommended to set/check correct File permissions and attributes instead. SERVER \u2014 The server name. sharename \u2014 The shared directory. mountpoint \u2014 The local directory where the share will be mounted. [-o options] \u2014 See for more information. Note : Abstain from using a trailing /. //SERVER/sharename/ will not work. If your mount does not work stable, stutters or freezes, try to enable different SMB protocol version with vers= option. For example, vers=2.0 for Windows Vista mount. If having timeouts on a mounted network share with cifs on a shutdown, see wpa_supplicant#Problem with mounted network shares (cifs) and shutdown. Storing share passwords Storing passwords in a world readable file is not recommended. A safer method is to use a credentials file instead, e.g. inside /etc/samba/credentials : /etc/samba/credentials/share username=myuser password=mypass For the mount command replace username=myuser,password=mypass with credentials=/etc/samba/credentials/share . The credential file should explicitly readable/writeable to root: # chown root:root /etc/samba/credentials # chmod 700 /etc/samba/credentials # chmod 600 /etc/samba/credentials/share Automatic mounting Note : You may need to enable systemd-networkd-wait-online.service or NetworkManager-wait-online.service (depending on your setup) to proper enable booting on start-up. Using NetworkManager and GIO/gvfs NetworkManager can be configured to run a script on network status change. This script uses the gio command so that it mounts the Samba shares automatically, the same way your file manager does, as explained below . The script also safely unmounts the Samba shares before the relevant network connection is disabled by listening for the and events. Make the script is executable after creating it. /etc/NetworkManager/dispatcher.d/30-samba.sh #!/bin/sh # Find the connection UUID with \"nmcli con show\" in terminal. # All NetworkManager connection types are supported: wireless, VPN, wired... WANTED_CON_UUID=\"CHANGE-ME-NOW-9c7eff15-010a-4b1c-a786-9b4efa218ba9\" # The user the share will be mounted under USER=\"yourusername\" # The path that appears in your file manager when you manually mount the share you want SMB_URL=\"smb://servername/share\" # Get runtime user directory. If it does not exist, do nothing and just exit XDG_RUNTIME_DIR=$(loginctl show-user --property=RuntimePath --value \"$USER\") || exit 0 if [ \"$CONNECTION_UUID\" = \"$WANTED_CON_UUID\" ]; then # Script parameter $1: network interface name, not used # Script parameter $2: dispatched event case \"$2\" in \"up\"|\"vpn-up\") su $USER -c \"DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus gio mount $SMB_URL\" ;; \"pre-down\"|\"vpn-pre-down\") su $USER -c \"DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus gio mount -uf $SMB_URL\" ;; esac fi Create a symlink inside to catch the events: # ln -s /etc/NetworkManager/dispatcher.d/30-samba.sh /etc/NetworkManager/dispatcher.d/pre-down.d/30-samba.sh As mount entry This is a simple example of a mount entry that requires authentication: As systemd unit Create a new file inside , e.g. . See for details. path to share path to mount the share share mounting options To use , start the unit and enable it to run on system boot. automount To automatically mount a share (when accessed, like autofs), one may use the following automount unit: Disable / stop the unit, and enable / start to automount the share when the mount path is being accessed. smbnetfs First, check if you can see all the shares you are interested in mounting: $ smbtree -U remote_user If that does not work, find and modify the following line in accordingly: domain master = auto Now restart and . If everything works as expected, install . Then, add the following line to : user_allow_other Now copy the directory to your home directory: $ cp -a /etc/smbnetfs/.smb ~ Then create a link to : $ ln -sf /etc/samba/smb.conf ~/.smb/smb.conf If a username and a password are required to access some of the shared folders, edit to include one or more entries like this: It is also possible to add entries for specific hosts to be mounted by smbnetfs, if necessary. More details can be found in . If you are using the Dolphin or GNOME Files , you may want to add the following to to avoid \"Disk full\" errors as smbnetfs by default will report 0 bytes of free space: When you are done with the configuration, you need to run $ chmod 600 ~/.smb/smbnetfs.* Otherwise, smbnetfs complains about 'insecure config file permissions'. Finally, to mount your Samba network neighbourhood to a directory of your choice, call $ smbnetfs mount_point Daemon The Arch Linux package also maintains an additional system-wide operation mode for smbnetfs. To enable it, you need to make the said modifications in the directory . Then, you can start and/or enable the daemon as usual. The system-wide mount point is at . autofs See Autofs for information on the kernel-based automounter for Linux. File manager configuration GNOME Files, Nemo, Caja, Thunar and PCManFM In order to access samba shares through GNOME Files, Nemo, Caja, Thunar or PCManFM, install the package. Press and enter in the location bar to access your share. The mounted share is likely to be present at or in the filesystem. KDE KDE applications (like Dolphin) has the ability to browse Samba shares built in. Use the path to browse the files. If you want to access files from on non-KDE application, you can install . To use a GUI in the KDE System Settings, you will need to install the package. Other graphical environments There are a number of useful programs, but they may need to have packages created for them. This can be done with the Arch package build system. The good thing about these others is that they do not require a particular environment to be installed to support them, and so they bring along less baggage. LinNeighborhood, RUmba, xffm-samba plugin for Xffm are not available in the official repositories or the AUR. As they are not officially (or even unofficially supported), they may be obsolete and may not work at all. Tips and tricks Discovering network shares If nothing is known about other systems on the local network, and automated tools such as smbnetfs are not available, you can manually probe for Samba shares. First, install the and packages. Use nmap to scan your local network to find systems with TCP port 445 open, which is the port used by the SMB protocol. Note that you may need to use or set a custom ping scan type (e.g. ) because Windows systems are usually firewalled. The first result is another system; the second happens to be the client from where this scan was performed. Now you can connect to there IP addresses directly, but if you want to use NetBIOS host names, you can use to check for NetBIOS names. Note that this will not work if NetBIOS is disabled on the server. Regardless of the output, look for \\<20> , which shows the host with open services. Use to list which services are shared on these systems. You can use NetBIOS host name ( in this example) instead of IP when available. If prompted for a password, pressing enter should still display the list: Remote control of Windows computer Samba offers a set of tools for communication with Windows. These can be handy if access to a Windows computer through remote desktop is not an option, as shown by some examples. Send shutdown command with a comment: $ net rpc shutdown -C \"comment\" -I IPADDRESS -U USERNAME%PASSWORD A forced shutdown instead can be invoked by changing -C with comment to a single -f. For a restart, only add -r, followed by a -C or -f. Stop and start services: $ net rpc service stop SERVICENAME -I IPADDRESS -U USERNAME%PASSWORD To see all possible net rpc command: $ net rpc Troubleshooting Failed to start Samba SMB/CIFS server Possible solutions: Check on syntactic errors with . Set correct permissions for and restart : # chmod 0755 /var/cache/samba/msg Permission issues on SELinux SELinux not allow samba to access user home directories by default, to solve this, run: # setsebool -P samba_enable_home_dirs 1 Similarly, and make Samba has the ability to read or \"read and write\" all files. Permission issues on AppArmor If using a share path located outside of a home or usershares directory, whitelist it in . E.g.: No dialect specified on mount The client is using an unsupported SMB/CIFS version that is required by the server. See #Restrict protocols for better security for more information. Unable to overwrite files, permissions errors Possible solutions: Append the mount option to the entry . Add to the section of the server's . Windows clients keep asking for password even if Samba shares are created with guest permissions Set inside the section of : map to guest = Bad Password If you are still using Samba \\< 4.10.10, use instead of . Windows 7 connectivity problems - mount error(12): cannot allocate memory A known Windows 7 bug that causes \"mount error(12): cannot allocate memory\" on an otherwise perfect cifs share on the Linux end can be fixed by setting a few registry keys on the Windows box as follows: (set to ) (set to ) Alternatively, start Command Prompt in Admin Mode and execute the following: reg add \"HKLM\\SYSTEM\\CurrentControlSet\\Control\\Session Manager\\Memory Management\" /v \"LargeSystemCache\" /t REG_DWORD /d 1 /f reg add \"HKLM\\SYSTEM\\CurrentControlSet\\Services\\LanmanServer\\Parameters\" /v \"Size\" /t REG_DWORD /d 3 /f Do one of the following for the settings to take effect: Restart Windows Restart the Server service via services.msc From the Command Prompt run: 'net stop lanmanserver' and 'net start lanmanserver' - The server may automatically restart after stopping it. Original article . Windows 10 1709 and up connectivity problems - \"Windows cannot access\" 0x80004005 This error affects some machines running Windows 10 version 1709 and later. It is not related to SMB1 being disabled in this version but to the fact that Microsoft disabled insecure logons for guests on this version for some, but not others. To fix, open Group Policy Editor (). Navigate to Computer configuration\\administrative templates\\network\\Lanman Workstation > Enable insecure guest logons and enable it. Alternatively,change the following value in the registry: [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Services\\LanmanWorkstation\\Parameters] \"AllowInsecureGuestAuth\"=dword:1 Error: Failed to retrieve printer list: NT_STATUS_UNSUCCESSFUL If you are a home user and using samba purely for file sharing from a server or NAS, you are probably not interested in sharing printers through it. If so, you can prevent this error from occurring by adding the following lines to your : Restart the samba service, , and then check your logs: # cat /var/log/samba/smbd.log and the error should now no longer be appearing. Sharing a folder fails It means that while you are sharing a folder from Dolphin (file manager) and everything seems ok at first, after restarting Dolphin the share icon is gone from the shared folder, and also some output like this in terminal ( Konsole ) output: \u2018net usershare\u2019 returned error 255: net usershare: usershares are currently disabled To fix it, enable usershare as described in #Enable Usershares . \"Browsing\" network fails with \"Failed to retrieve share list from server\" And you are using a firewall (iptables) because you do not trust your local (school, university, hotel) network. This may be due to the following: When the smbclient is browsing the local network it sends out a broadcast request on udp port 137. The servers on the network then reply to your client but as the source address of this reply is different from the destination address iptables saw when sending the request for the listing out, iptables will not recognize the reply as being \"ESTABLISHED\" or \"RELATED\", and hence the packet is dropped. A possible solution is to add: iptables -t raw -A OUTPUT -p udp -m udp --dport 137 -j CT --helper netbios-ns to your iptables setup. For Uncomplicated Firewall , you need to add to the end of the following line in IPT_MODULES=\"nf_conntrack_ftp nf_nat_ftp nf_conntrack_irc nf_nat_irc\" and then run the following commands as root: echo 1 > /proc/sys/net/netfilter/nf_conntrack_helper ufw allow CIFS ufw reload To make this change persistent across reboots, add the following line at the end of : net.netfilter.nf_conntrack_helper=1 Protocol negotiation failed: NT_STATUS_INVALID_NETWORK_RESPONSE The client probably does not have access to shares. Make sure clients' IP address is in line in . Another problem could be, that the client uses an invalid protocol version. To check this try to connect with the where you specify the maximum protocol version manually: $ smbclient -U -L // -m -W If the command was successful then create a configuration file: Connection to SERVER failed: (Error NT_STATUS_UNSUCCESSFUL) You are probably passing a wrong server name to . To find out the server name, run on the server and look at \"Transient hostname\" line Connection to SERVER failed: (Error NT_STATUS_CONNECTION_REFUSED) Make sure that the server has started. The shared directories should exist and be accessible. Protocol negotiation failed: NT_STATUS_CONNECTION_RESET Probably the server is configured not to accept protocol SMB1. Add option in . Or just pass argument to . Password Error when correct credentials are given (error 1326) Samba 4.5 has NTLMv1 authentication disabled by default. It is recommend to install the latest available upgrades on clients and deny access for unsupported clients. If you still need support for very old clients without NTLMv2 support (e.g. Windows XP), it is possible force enable NTLMv1, although this is not recommend for security reasons: If NTLMv2 clients are unable to authenticate when NTLMv1 has been enabled, create the following file on the client: This change also affects samba shares mounted with mount.cifs . If after upgrade to Samba 4.5 your mount fails, add the sec=ntlmssp option to your mount command, e.g. mount.cifs //server/share /mnt/point -o sec=ntlmssp,... See the man page: ntlmssp - Use NTLMv2 password hashing encapsulated in Raw NTLMSSP message. The default in mainline kernel versions prior to v3.8 was sec=ntlm . In v3.8, the default was changed to sec=ntlmssp . Mapping reserved Windows characters Starting with kernel 3.18, the cifs module uses the \"mapposix\" option by default . When mounting a share using unix extensions and a default Samba configuration, files and directories containing one of the seven reserved Windows characters are listed but cannot be accessed. Possible solutions are: Use the undocumented mount option for cifs # mount.cifs //server/share /mnt/point -o nomapposix Configure Samba to remap (\"SFM\", Services for Mac) style characters to the correct native ones using fruit Manually remap forbidden characters using catia The latter approach (using catia or fruit) has the drawback of filtering files with unprintable characters. Folder shared inside graphical environment is not available to guests This section presupposes: Usershares are configured following previous section A shared folder has been created as a non-root user from GUI Guests access has been set to shared folder during creation Samba service has been restarted at least once since last file modification For clarification purpose only, in the following sub-sections is assumed: Shared folder is located inside user home directory path () Shared folder name is MySharedFiles Guest access is read-only. Windows users will access shared folder content without login prompt Verify correct samba configuration Run the following command from a terminal to test configuration file correctness: $ testparm Verify correct shared folder creation Run the following commands from a terminal: $ cd /var/lib/samba/usershares $ ls If everything is fine, you will notice a file named Read the file contents using the following command: $ cat mysharedfiles The terminal output should display something like this: Verify folder access by guest Run the following command from a terminal. If prompted for a password, just press Enter: $ smbclient -L localhost If everything is fine, MySharedFiles should be displayed under column Run the following command in order to access the shared folder as guest (anonymous login) $ smbclient -N //localhost/MySharedFiles If everything is fine samba client prompt will be displayed: smb: \\> From samba prompt verify guest can list directory contents: smb: \\> ls If the error is displayed, the issue is likely to be with Unix directory permissions. Ensure that your samba user has access to the folder and all parent folders. You can test this by sudoing to the user and attempting to list the mount directory, and all of its parents. Mount error: Host is down This error might be seen when mounting shares of Synology NAS servers. Use the mount option to solve it. Software caused connection abort File managers that utilizes can show the error when writing a file to a share/server. This may be due to the server running SMB/CIFS version 1, which many routers use for USB drive sharing (e.g. Belkin routers). To write to these shares specify the CIFS version with the option . E.g.: This can also happen after updating Samba to version 4.11, which deactivates SMB1 as default, and accessing any Samba share. You can reenable it by adding Connection problem (due to authentification error) Be sure that you do not leave any space characters before your username in Samba client configuration file as follows: The correct format is: Windows 1709 or up does not discover the samba server in Network view With Windows 10 version 1511, support for SMBv1 and thus NetBIOS device discovery was disabled by default. Depending on the actual edition, later versions of Windows starting from version 1709 (\"Fall Creators Update\") do not allow the installation of the SMBv1 client anymore. This causes hosts running Samba not to be listed in the Explorer's \"Network (Neighborhood)\" views. While there is no connectivity problem and Samba will still run fine, users might want to have their Samba hosts to be listed by Windows automatically. implements a Web Service Discovery host daemon. This enables (Samba) hosts, like your local NAS device, to be found by Web Service Discovery Clients like Windows. The default settings should work for most installations, all you need to do is start enable . If the default configuration (advertise itself as the machine hostname in group \"WORKGROUP\") should be all you need in most cases. If you need, you can change configuration options by passing additional arguments to wsdd by adding them in (see the manual page for wsdd for details). does the same thing, but is written in C instead of Python. By default, it will look for the and values in . IOS Files can no longer copy-to Samba share on Arch Linux beginning with IOS 14.5 Beginning with IOS 14.5 attempting to transfer from a device running IOS using the \"Files\" app to a samba share on Arch Linux will result in the error: The operation couldn't be completed Operation canceled To correct this problem, add add the following to the global section of your and restart . Comment optional: ## addition for IOS Files transfer-to server vfs object = fruit streams_xattr See https://apple.stackexchange.com/q/424681 Apple.Stackexchange.com - \"The operation couldn't be completed\"/\"Operation canceled\" error message when saving to a Samba share via Files app. Slow initial connections from certain clients without other performance problems Some SMB clients, such as Solid Explorer for Android, take significantly longer to connect to Samba if they fail to resolve the NetBIOS name. Enabling will greatly speed up initial connections if this is the case. Since this is a bug in the client software, please report such cases to the authors of conflicting software. See also Official website Samba: An Introduction Samba 3.2.x HOWTO and Reference Guide (outdated but still most extensive documentation) Wikipedia Gentoo:Samba/Guide Debian:Samba/ServerSimple KSMBD - A linux kernel server which implements SMB3 protocol in kernel space for sharing files over network. Category:Network sharing Category:Servers","title":"Samba"},{"location":"smb/#samba","text":"Samba is the standard Windows interoperability suite of programs for Linux and Unix. Since 1992, Samba has provided secure, stable and fast file and print services for all clients using the SMB/CIFS protocol, such as all versions of DOS and Windows, OS/2, Linux and many others. To share files through Samba, see #Server section; to access files shared through Samba on other machines, please see #Client section.","title":"Samba"},{"location":"smb/#server","text":"cd /mnt/ sudo mkdir media nano /home/user/.credentials sudo chmod 400 /home/user/.credentials sudo mount -t cifs -o rw,vers=3.0,credentials=/home/user/.credentials //coco.local/media/ /mnt/media sudo nano /etc/fstab # smbclient -L hostname -U% # //coco.local/media /mnt/media cifs vers=3.0,credentials=/home/user/.credentials","title":"Server"},{"location":"smb/#installation","text":"Install the package. Samba is configured in the configuration file, which is extensively documented in . Because the package does not provide this file, one needs to create it before starting . A documented example as in from the Samba git repository may be used to setup .","title":"Installation"},{"location":"smb/#enabling-and-starting-services","text":"To provide basic file sharing through SMB, enable/start . See for details. If you want to make your server accessible via NetBIOS host name, set the desired name in the option in and enable/start . See for details.","title":"Enabling and starting services"},{"location":"smb/#make-the-server-discoverable","text":"Install the package, then enable/start to make the samba server discoverable with Zeroconf . It should work for most non-Windows file managers (macOS Finder, various GUI-based file managers on Linux & BSD etc.) If is not running, the server will still be accessible, just not discoverable, i.e. it will not show up in file managers, but you can still connect to the server directly by IP or domain. Windows Explorer relies on the WS-Discovery protocol instead; see #Windows 1709 or up does not discover the samba server in Network view .","title":"Make the server discoverable"},{"location":"smb/#configure-firewall","text":"If you are using a firewall , do not forget to open required ports (usually 137-139 + 445). For a complete list, see Samba port usage .","title":"Configure firewall"},{"location":"smb/#ufw-rule","text":"A Ufw App Profile for SMB/CIFS is included by default with the default installation of UFW in . Allow Samba by running as root. If you deleted the profile, create/edit and add the following content: [Samba] title=LanManager-like file and printer server for Unix description=The Samba software suite is a collection of programs that implements the SMB/CIFS protocol for unix systems, allowing you to serve files and printers to Windows, NT, OS/2 and DOS clients. This protocol is sometimes also referred to as the LanManager or NetBIOS protocol. ports=137,138/udp|139,445/tcp Then load the profile into UFW run as root. Then finally, allow Samba by running as root.","title":"UFW Rule"},{"location":"smb/#firewalld-service","text":"To configure firewalld to allow Samba in the home zone, run: # firewall-cmd --permanent --add-service={samba,samba-client,samba-dc} --zone=home The three services listed are: : for sharing files with others. : to browse shares on other machines on the network. : for Samba/Active Directory domain controller . ensures the changes remain after is restarted .","title":"firewalld service"},{"location":"smb/#basic-configuration","text":"","title":"Basic configuration"},{"location":"smb/#user-management","text":"The following section describes creating a local (tdbsam) database of Samba users. For user authentication and other purposes, Samba can also be bound to an Active Directory domain, can itself serve as an Active Directory domain controller, or can be used with an LDAP server.","title":"User management"},{"location":"smb/#adding-a-user","text":"Samba requires a Linux user account - you may use an existing user account or create a new one . Although the user name is shared with Linux system, Samba uses a password separate from that of the Linux user accounts. Replace with the chosen Samba user account: # smbpasswd -a samba_user Depending on the server role , existing File permissions and attributes may need to be altered for the Samba user account. If you want the new user only to be allowed to remotely access the file server shares through Samba, you can restrict other login options\uff1a disabling shell - disabling SSH logons - edit , change option Also see Security for hardening your system.","title":"Adding a user"},{"location":"smb/#listing-users","text":"Samba users can be listed using the command: # pdbedit -L -v","title":"Listing users"},{"location":"smb/#changing-user-password","text":"To change a user password, use : # smbpasswd samba_user","title":"Changing user password"},{"location":"smb/#creating-an-anonymous-share","text":"1. Create a Linux user which anonymous Samba users will be mapped to. # useradd guest -s /bin/nologin 2. Add the following to : Anonymous users will now be mapped to the Linux user and have the ability to access any directories defined in , which is configured to be in the example above. Make sure that the Linux user has the proper permissions to access files in . Also, make sure shares have been properly defined as per the Share Definitions section of smb.conf.default .","title":"Creating an anonymous share"},{"location":"smb/#advanced-configuration","text":"","title":"Advanced configuration"},{"location":"smb/#enable-symlink-following","text":"Then, restart .","title":"Enable symlink following"},{"location":"smb/#enable-server-side-copy-for-macos-clients","text":"Server-side copy eliminates the need to transfer data between the server and the client when copying files on the server. This is enabled by default, but it doesn't work with macOS clients. If you have macOS clients, you need to add the following configuration to and then restart .","title":"Enable server-side copy for macOS clients"},{"location":"smb/#enable-usershares","text":"Usershares is a feature that gives non-root users the capability to add, modify, and delete their own share definitions. See . Create a directory for usershares: Create a user group : Change the owner of the directory to and the group to : Change the permissions of the directory so that users in the group can create files. This command also sets sticky bit , which is important to prevent users from deleting usershares of other users: Set the following parameters in the configuration file: Add the user to the sambashare group. Replace with the name of your user: # gpasswd sambashare -a your_username Restart and services. Log out and log back in. If you want to share paths inside your home directory you must make it accessible for the group others . In the GUI, you can use Thunar or Dolphin - right click on any directory and share it on the network. In the CLI, use one of the following commands, replacing italic sharename , user , ... : # net usershare add sharename abspath [ comment ] [ user :{R|D|F}] [guest_ok={y|n}] # net usershare delete sharename # net usershare list wildcard-sharename # net usershare info wildcard-sharename","title":"Enable Usershares"},{"location":"smb/#set-and-forcing-permissions","text":"Permissions may be applied to both the server and shares: See for a full overview of possible permission flags and settings.","title":"Set and forcing permissions"},{"location":"smb/#restrict-protocols-for-better-security","text":"Append and in to force usage of a minimum and maximum protocol: See in for an overview of supported protocols. For compatibility with older clients and/or servers, you might need to set or , but please note that this makes you vulnerable to exploits in SMB1 including ransomware attacks. Clients using may need to specify the correct , e.g.: # mount -t cifs // SERVER / sharename /mnt/ mountpoint -o username= username ,password= password ,iocharset= utf8 ,vers= 3.1.1 See for more information.","title":"Restrict protocols for better security"},{"location":"smb/#use-native-smb-transport-encryption","text":"Native SMB transport encryption is available in SMB version 3.0 or newer. Clients supporting this type of encryption include Windows 8 and newer, Windows server 2012 and newer, and smbclient of Samba 4.1 and newer. To use native SMB transport encryption by default, set the parameter globally and/or by share. Possible values are , (default value), , or : To configure encryption for on the client side, use the option . See for more information, especially the paragraphs Effects for SMB1 and Effects for SMB2 .","title":"Use native SMB transport encryption"},{"location":"smb/#disable-printer-sharing","text":"By default Samba shares printers configured using CUPS . If you do not want printers to be shared, use the following settings:","title":"Disable printer sharing"},{"location":"smb/#block-certain-file-extensions-on-samba-share","text":"Samba offers an option to block files with certain patterns, like file extensions. This option can be used to prevent dissemination of viruses or to dissuade users from wasting space with certain files. More information about this option can be found in .","title":"Block certain file extensions on Samba share"},{"location":"smb/#improve-throughput","text":"The default settings should be sufficient for most users. However setting the 'socket options' correct can improve performance, but getting them wrong can degrade it by just as much. Test the effect before making any large changes. Read the man page before applying any of the options listed below. The following settings should be appended to the section of . Setting a deadtime is useful to stop a server's resources from being exhausted by a large number of inactive connections: deadtime = 30 The usage of sendfile may make more efficient use of the system CPU's and cause Samba to be faster: use sendfile = yes Setting min receivefile size allows zero-copy writes directly from network socket buffers into the filesystem buffer cache (if available). It may improve performance but user testing is recommended: min receivefile size = 16384 Increasing the receive/send buffers size and socket optimize flags might be useful to improve throughput. It is recommended to test each flag separately as it may cause issues on some networks: socket options = IPTOS_LOWDELAY TCP_NODELAY IPTOS_THROUGHPUT SO_RCVBUF=131072 SO_SNDBUF=131072","title":"Improve throughput"},{"location":"smb/#enable-access-for-old-clientsdevices","text":"Latest versions of Samba no longer offer older authentication methods and protocols which are still used by some older clients (IP cameras, etc). These devices usually require Samba server to allow NTMLv1 authentication and NT1 version of the protocol, known as CIFS. For these devices to work with latest Samba, you need to add these two configuration parameters into section: server min protocol = NT1 ntlm auth = yes Anonymous/guest access to a share requires just the first parameter. If the old device will access with username and password, you also need the add the second line too.","title":"Enable access for old clients/devices"},{"location":"smb/#enable-spotlight-searching","text":"Spotlight allows supporting clients (e.g. MacOS Finder) to quickly search shared files. Install and start/enable OpenSearch . Install , configure the directories you want to index in , and start/enable for periodic indexing. Edit as described in the Samba wiki to enable Spotlight per share, and restart to apply the changes.","title":"Enable Spotlight searching"},{"location":"smb/#client","text":"Install for an -like command line interface. See for commonly used commands. For a lightweight alternative (without support for listing public shares, etc.), install that provides . Depending on the desktop environment , GUI methods may be available. See #File manager configuration for use with a file manager.","title":"Client"},{"location":"smb/#list-public-shares","text":"The following command lists public shares on a server: $ smbclient -L hostname -U% Alternatively, running will show a tree diagram of all the shares. It uses broadcast queries and is therefore not advisable on a network with a lot of computers, but can be helpful for diagnosing if you have the correct sharename. The () option suppresses the password prompt.","title":"List public shares"},{"location":"smb/#netbioswins-host-names","text":"Samba clients handle NetBIOS host names automatically by default (the behavior is controlled by the option in ). Other programs (including ) typically use Name Service Switch , which does not handle NetBIOS by default. The package provides a libnss driver to resolve NetBIOS host names. To use it, install it along with the package (which provides the winbindd daemon), start/enable and add to the line in : Now, during host resolving (e.g. when using or just ), winbindd will resolve the host name by sending queries using NetBIOS Name Service (NBNS, also known as WINS) protocol. By default it sends a broadcast query to your local network. If you have a WINS server, you can add to and restart , then winbindd and other Samba clients will send unicast queries to the specified IP. If you want to resolve your local host name (specified in the option in ), start/enable , which will handle incoming queries. You can test WINS resolution with . By default it sends broadcast queries to your local network regardless of the option. Note that WINS resolution requires incoming traffic originating from port 137.","title":"NetBIOS/WINS host names"},{"location":"smb/#disable-netbioswins-support","text":"When not using NetBIOS/WINS host name resolution, it may be preferred to disable this protocol: /etc/samba/smb.conf [global] disable netbios = yes dns proxy = no Finally disable / stop .","title":"Disable NetBIOS/WINS support"},{"location":"smb/#manual-mounting","text":"Mount the share using as . Not all the options listed below are needed or desirable: # mount --mkdir -t cifs // SERVER / sharename /mnt/ mountpoint -o username= username ,password= password ,workgroup= workgroup ,iocharset= utf8 ,uid= username ,gid= group The options uid and gid corresponds to the local (e.g. client) user / user group to have read/write access on the given path. Note : If the uid and gid being used does not match the user of the server, the forceuid and forcegid options may be helpful. However note permissions assigned to a file when forceuid or forcegid are in effect may not reflect the the real (server) permissions. See the File And Directory Ownership And Permissions section in mount.cifs(8) \u00a7\u202fFILE AND DIRECTORY OWNERSHIP AND PERMISSIONS for more information. To mount a Windows share without authentication, use \"username=*\". Warning : Using uid and/or gid as mount options may cause I/O errors, it is recommended to set/check correct File permissions and attributes instead. SERVER \u2014 The server name. sharename \u2014 The shared directory. mountpoint \u2014 The local directory where the share will be mounted. [-o options] \u2014 See for more information. Note : Abstain from using a trailing /. //SERVER/sharename/ will not work. If your mount does not work stable, stutters or freezes, try to enable different SMB protocol version with vers= option. For example, vers=2.0 for Windows Vista mount. If having timeouts on a mounted network share with cifs on a shutdown, see wpa_supplicant#Problem with mounted network shares (cifs) and shutdown.","title":"Manual mounting"},{"location":"smb/#storing-share-passwords","text":"Storing passwords in a world readable file is not recommended. A safer method is to use a credentials file instead, e.g. inside /etc/samba/credentials : /etc/samba/credentials/share username=myuser password=mypass For the mount command replace username=myuser,password=mypass with credentials=/etc/samba/credentials/share . The credential file should explicitly readable/writeable to root: # chown root:root /etc/samba/credentials # chmod 700 /etc/samba/credentials # chmod 600 /etc/samba/credentials/share","title":"Storing share passwords"},{"location":"smb/#automatic-mounting","text":"Note : You may need to enable systemd-networkd-wait-online.service or NetworkManager-wait-online.service (depending on your setup) to proper enable booting on start-up.","title":"Automatic mounting"},{"location":"smb/#using-networkmanager-and-giogvfs","text":"NetworkManager can be configured to run a script on network status change. This script uses the gio command so that it mounts the Samba shares automatically, the same way your file manager does, as explained below . The script also safely unmounts the Samba shares before the relevant network connection is disabled by listening for the and events. Make the script is executable after creating it. /etc/NetworkManager/dispatcher.d/30-samba.sh #!/bin/sh # Find the connection UUID with \"nmcli con show\" in terminal. # All NetworkManager connection types are supported: wireless, VPN, wired... WANTED_CON_UUID=\"CHANGE-ME-NOW-9c7eff15-010a-4b1c-a786-9b4efa218ba9\" # The user the share will be mounted under USER=\"yourusername\" # The path that appears in your file manager when you manually mount the share you want SMB_URL=\"smb://servername/share\" # Get runtime user directory. If it does not exist, do nothing and just exit XDG_RUNTIME_DIR=$(loginctl show-user --property=RuntimePath --value \"$USER\") || exit 0 if [ \"$CONNECTION_UUID\" = \"$WANTED_CON_UUID\" ]; then # Script parameter $1: network interface name, not used # Script parameter $2: dispatched event case \"$2\" in \"up\"|\"vpn-up\") su $USER -c \"DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus gio mount $SMB_URL\" ;; \"pre-down\"|\"vpn-pre-down\") su $USER -c \"DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus gio mount -uf $SMB_URL\" ;; esac fi Create a symlink inside to catch the events: # ln -s /etc/NetworkManager/dispatcher.d/30-samba.sh /etc/NetworkManager/dispatcher.d/pre-down.d/30-samba.sh","title":"Using NetworkManager and GIO/gvfs"},{"location":"smb/#as-mount-entry","text":"This is a simple example of a mount entry that requires authentication:","title":"As mount entry"},{"location":"smb/#as-systemd-unit","text":"Create a new file inside , e.g. . See for details. path to share path to mount the share share mounting options To use , start the unit and enable it to run on system boot.","title":"As systemd unit"},{"location":"smb/#automount","text":"To automatically mount a share (when accessed, like autofs), one may use the following automount unit: Disable / stop the unit, and enable / start to automount the share when the mount path is being accessed.","title":"automount"},{"location":"smb/#smbnetfs","text":"First, check if you can see all the shares you are interested in mounting: $ smbtree -U remote_user If that does not work, find and modify the following line in accordingly: domain master = auto Now restart and . If everything works as expected, install . Then, add the following line to : user_allow_other Now copy the directory to your home directory: $ cp -a /etc/smbnetfs/.smb ~ Then create a link to : $ ln -sf /etc/samba/smb.conf ~/.smb/smb.conf If a username and a password are required to access some of the shared folders, edit to include one or more entries like this: It is also possible to add entries for specific hosts to be mounted by smbnetfs, if necessary. More details can be found in . If you are using the Dolphin or GNOME Files , you may want to add the following to to avoid \"Disk full\" errors as smbnetfs by default will report 0 bytes of free space: When you are done with the configuration, you need to run $ chmod 600 ~/.smb/smbnetfs.* Otherwise, smbnetfs complains about 'insecure config file permissions'. Finally, to mount your Samba network neighbourhood to a directory of your choice, call $ smbnetfs mount_point","title":"smbnetfs"},{"location":"smb/#daemon","text":"The Arch Linux package also maintains an additional system-wide operation mode for smbnetfs. To enable it, you need to make the said modifications in the directory . Then, you can start and/or enable the daemon as usual. The system-wide mount point is at .","title":"Daemon"},{"location":"smb/#autofs","text":"See Autofs for information on the kernel-based automounter for Linux.","title":"autofs"},{"location":"smb/#file-manager-configuration","text":"","title":"File manager configuration"},{"location":"smb/#gnome-files-nemo-caja-thunar-and-pcmanfm","text":"In order to access samba shares through GNOME Files, Nemo, Caja, Thunar or PCManFM, install the package. Press and enter in the location bar to access your share. The mounted share is likely to be present at or in the filesystem.","title":"GNOME Files, Nemo, Caja, Thunar and PCManFM"},{"location":"smb/#kde","text":"KDE applications (like Dolphin) has the ability to browse Samba shares built in. Use the path to browse the files. If you want to access files from on non-KDE application, you can install . To use a GUI in the KDE System Settings, you will need to install the package.","title":"KDE"},{"location":"smb/#other-graphical-environments","text":"There are a number of useful programs, but they may need to have packages created for them. This can be done with the Arch package build system. The good thing about these others is that they do not require a particular environment to be installed to support them, and so they bring along less baggage. LinNeighborhood, RUmba, xffm-samba plugin for Xffm are not available in the official repositories or the AUR. As they are not officially (or even unofficially supported), they may be obsolete and may not work at all.","title":"Other graphical environments"},{"location":"smb/#tips-and-tricks","text":"","title":"Tips and tricks"},{"location":"smb/#discovering-network-shares","text":"If nothing is known about other systems on the local network, and automated tools such as smbnetfs are not available, you can manually probe for Samba shares. First, install the and packages. Use nmap to scan your local network to find systems with TCP port 445 open, which is the port used by the SMB protocol. Note that you may need to use or set a custom ping scan type (e.g. ) because Windows systems are usually firewalled. The first result is another system; the second happens to be the client from where this scan was performed. Now you can connect to there IP addresses directly, but if you want to use NetBIOS host names, you can use to check for NetBIOS names. Note that this will not work if NetBIOS is disabled on the server. Regardless of the output, look for \\<20> , which shows the host with open services. Use to list which services are shared on these systems. You can use NetBIOS host name ( in this example) instead of IP when available. If prompted for a password, pressing enter should still display the list:","title":"Discovering network shares"},{"location":"smb/#remote-control-of-windows-computer","text":"Samba offers a set of tools for communication with Windows. These can be handy if access to a Windows computer through remote desktop is not an option, as shown by some examples. Send shutdown command with a comment: $ net rpc shutdown -C \"comment\" -I IPADDRESS -U USERNAME%PASSWORD A forced shutdown instead can be invoked by changing -C with comment to a single -f. For a restart, only add -r, followed by a -C or -f. Stop and start services: $ net rpc service stop SERVICENAME -I IPADDRESS -U USERNAME%PASSWORD To see all possible net rpc command: $ net rpc","title":"Remote control of Windows computer"},{"location":"smb/#troubleshooting","text":"","title":"Troubleshooting"},{"location":"smb/#failed-to-start-samba-smbcifs-server","text":"Possible solutions: Check on syntactic errors with . Set correct permissions for and restart : # chmod 0755 /var/cache/samba/msg","title":"Failed to start Samba SMB/CIFS server"},{"location":"smb/#permission-issues-on-selinux","text":"SELinux not allow samba to access user home directories by default, to solve this, run: # setsebool -P samba_enable_home_dirs 1 Similarly, and make Samba has the ability to read or \"read and write\" all files.","title":"Permission issues on SELinux"},{"location":"smb/#permission-issues-on-apparmor","text":"If using a share path located outside of a home or usershares directory, whitelist it in . E.g.:","title":"Permission issues on AppArmor"},{"location":"smb/#no-dialect-specified-on-mount","text":"The client is using an unsupported SMB/CIFS version that is required by the server. See #Restrict protocols for better security for more information.","title":"No dialect specified on mount"},{"location":"smb/#unable-to-overwrite-files-permissions-errors","text":"Possible solutions: Append the mount option to the entry . Add to the section of the server's .","title":"Unable to overwrite files, permissions errors"},{"location":"smb/#windows-clients-keep-asking-for-password-even-if-samba-shares-are-created-with-guest-permissions","text":"Set inside the section of : map to guest = Bad Password If you are still using Samba \\< 4.10.10, use instead of .","title":"Windows clients keep asking for password even if Samba shares are created with guest permissions"},{"location":"smb/#windows-7-connectivity-problems-mount-error12-cannot-allocate-memory","text":"A known Windows 7 bug that causes \"mount error(12): cannot allocate memory\" on an otherwise perfect cifs share on the Linux end can be fixed by setting a few registry keys on the Windows box as follows: (set to ) (set to ) Alternatively, start Command Prompt in Admin Mode and execute the following: reg add \"HKLM\\SYSTEM\\CurrentControlSet\\Control\\Session Manager\\Memory Management\" /v \"LargeSystemCache\" /t REG_DWORD /d 1 /f reg add \"HKLM\\SYSTEM\\CurrentControlSet\\Services\\LanmanServer\\Parameters\" /v \"Size\" /t REG_DWORD /d 3 /f Do one of the following for the settings to take effect: Restart Windows Restart the Server service via services.msc From the Command Prompt run: 'net stop lanmanserver' and 'net start lanmanserver' - The server may automatically restart after stopping it. Original article .","title":"Windows 7 connectivity problems - mount error(12): cannot allocate memory"},{"location":"smb/#windows-10-1709-and-up-connectivity-problems-windows-cannot-access-0x80004005","text":"This error affects some machines running Windows 10 version 1709 and later. It is not related to SMB1 being disabled in this version but to the fact that Microsoft disabled insecure logons for guests on this version for some, but not others. To fix, open Group Policy Editor (). Navigate to Computer configuration\\administrative templates\\network\\Lanman Workstation > Enable insecure guest logons and enable it. Alternatively,change the following value in the registry: [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Services\\LanmanWorkstation\\Parameters] \"AllowInsecureGuestAuth\"=dword:1","title":"Windows 10 1709 and up connectivity problems - \"Windows cannot access\" 0x80004005"},{"location":"smb/#error-failed-to-retrieve-printer-list-nt_status_unsuccessful","text":"If you are a home user and using samba purely for file sharing from a server or NAS, you are probably not interested in sharing printers through it. If so, you can prevent this error from occurring by adding the following lines to your : Restart the samba service, , and then check your logs: # cat /var/log/samba/smbd.log and the error should now no longer be appearing.","title":"Error: Failed to retrieve printer list: NT_STATUS_UNSUCCESSFUL"},{"location":"smb/#sharing-a-folder-fails","text":"It means that while you are sharing a folder from Dolphin (file manager) and everything seems ok at first, after restarting Dolphin the share icon is gone from the shared folder, and also some output like this in terminal ( Konsole ) output: \u2018net usershare\u2019 returned error 255: net usershare: usershares are currently disabled To fix it, enable usershare as described in #Enable Usershares .","title":"Sharing a folder fails"},{"location":"smb/#browsing-network-fails-with-failed-to-retrieve-share-list-from-server","text":"And you are using a firewall (iptables) because you do not trust your local (school, university, hotel) network. This may be due to the following: When the smbclient is browsing the local network it sends out a broadcast request on udp port 137. The servers on the network then reply to your client but as the source address of this reply is different from the destination address iptables saw when sending the request for the listing out, iptables will not recognize the reply as being \"ESTABLISHED\" or \"RELATED\", and hence the packet is dropped. A possible solution is to add: iptables -t raw -A OUTPUT -p udp -m udp --dport 137 -j CT --helper netbios-ns to your iptables setup. For Uncomplicated Firewall , you need to add to the end of the following line in IPT_MODULES=\"nf_conntrack_ftp nf_nat_ftp nf_conntrack_irc nf_nat_irc\" and then run the following commands as root: echo 1 > /proc/sys/net/netfilter/nf_conntrack_helper ufw allow CIFS ufw reload To make this change persistent across reboots, add the following line at the end of : net.netfilter.nf_conntrack_helper=1","title":"\"Browsing\" network fails with \"Failed to retrieve share list from server\""},{"location":"smb/#protocol-negotiation-failed-nt_status_invalid_network_response","text":"The client probably does not have access to shares. Make sure clients' IP address is in line in . Another problem could be, that the client uses an invalid protocol version. To check this try to connect with the where you specify the maximum protocol version manually: $ smbclient -U -L // -m -W If the command was successful then create a configuration file:","title":"Protocol negotiation failed: NT_STATUS_INVALID_NETWORK_RESPONSE"},{"location":"smb/#connection-to-server-failed-error-nt_status_unsuccessful","text":"You are probably passing a wrong server name to . To find out the server name, run on the server and look at \"Transient hostname\" line","title":"Connection to SERVER failed: (Error NT_STATUS_UNSUCCESSFUL)"},{"location":"smb/#connection-to-server-failed-error-nt_status_connection_refused","text":"Make sure that the server has started. The shared directories should exist and be accessible.","title":"Connection to SERVER failed: (Error NT_STATUS_CONNECTION_REFUSED)"},{"location":"smb/#protocol-negotiation-failed-nt_status_connection_reset","text":"Probably the server is configured not to accept protocol SMB1. Add option in . Or just pass argument to .","title":"Protocol negotiation failed: NT_STATUS_CONNECTION_RESET"},{"location":"smb/#password-error-when-correct-credentials-are-given-error-1326","text":"Samba 4.5 has NTLMv1 authentication disabled by default. It is recommend to install the latest available upgrades on clients and deny access for unsupported clients. If you still need support for very old clients without NTLMv2 support (e.g. Windows XP), it is possible force enable NTLMv1, although this is not recommend for security reasons: If NTLMv2 clients are unable to authenticate when NTLMv1 has been enabled, create the following file on the client: This change also affects samba shares mounted with mount.cifs . If after upgrade to Samba 4.5 your mount fails, add the sec=ntlmssp option to your mount command, e.g. mount.cifs //server/share /mnt/point -o sec=ntlmssp,... See the man page: ntlmssp - Use NTLMv2 password hashing encapsulated in Raw NTLMSSP message. The default in mainline kernel versions prior to v3.8 was sec=ntlm . In v3.8, the default was changed to sec=ntlmssp .","title":"Password Error when correct credentials are given (error 1326)"},{"location":"smb/#mapping-reserved-windows-characters","text":"Starting with kernel 3.18, the cifs module uses the \"mapposix\" option by default . When mounting a share using unix extensions and a default Samba configuration, files and directories containing one of the seven reserved Windows characters are listed but cannot be accessed. Possible solutions are: Use the undocumented mount option for cifs # mount.cifs //server/share /mnt/point -o nomapposix Configure Samba to remap (\"SFM\", Services for Mac) style characters to the correct native ones using fruit Manually remap forbidden characters using catia The latter approach (using catia or fruit) has the drawback of filtering files with unprintable characters.","title":"Mapping reserved Windows characters"},{"location":"smb/#folder-shared-inside-graphical-environment-is-not-available-to-guests","text":"This section presupposes: Usershares are configured following previous section A shared folder has been created as a non-root user from GUI Guests access has been set to shared folder during creation Samba service has been restarted at least once since last file modification For clarification purpose only, in the following sub-sections is assumed: Shared folder is located inside user home directory path () Shared folder name is MySharedFiles Guest access is read-only. Windows users will access shared folder content without login prompt","title":"Folder shared inside graphical environment is not available to guests"},{"location":"smb/#verify-correct-samba-configuration","text":"Run the following command from a terminal to test configuration file correctness: $ testparm","title":"Verify correct samba configuration"},{"location":"smb/#verify-correct-shared-folder-creation","text":"Run the following commands from a terminal: $ cd /var/lib/samba/usershares $ ls If everything is fine, you will notice a file named Read the file contents using the following command: $ cat mysharedfiles The terminal output should display something like this:","title":"Verify correct shared folder creation"},{"location":"smb/#verify-folder-access-by-guest","text":"Run the following command from a terminal. If prompted for a password, just press Enter: $ smbclient -L localhost If everything is fine, MySharedFiles should be displayed under column Run the following command in order to access the shared folder as guest (anonymous login) $ smbclient -N //localhost/MySharedFiles If everything is fine samba client prompt will be displayed: smb: \\> From samba prompt verify guest can list directory contents: smb: \\> ls If the error is displayed, the issue is likely to be with Unix directory permissions. Ensure that your samba user has access to the folder and all parent folders. You can test this by sudoing to the user and attempting to list the mount directory, and all of its parents.","title":"Verify folder access by guest"},{"location":"smb/#mount-error-host-is-down","text":"This error might be seen when mounting shares of Synology NAS servers. Use the mount option to solve it.","title":"Mount error: Host is down"},{"location":"smb/#software-caused-connection-abort","text":"File managers that utilizes can show the error when writing a file to a share/server. This may be due to the server running SMB/CIFS version 1, which many routers use for USB drive sharing (e.g. Belkin routers). To write to these shares specify the CIFS version with the option . E.g.: This can also happen after updating Samba to version 4.11, which deactivates SMB1 as default, and accessing any Samba share. You can reenable it by adding","title":"Software caused connection abort"},{"location":"smb/#connection-problem-due-to-authentification-error","text":"Be sure that you do not leave any space characters before your username in Samba client configuration file as follows: The correct format is:","title":"Connection problem (due to authentification error)"},{"location":"smb/#windows-1709-or-up-does-not-discover-the-samba-server-in-network-view","text":"With Windows 10 version 1511, support for SMBv1 and thus NetBIOS device discovery was disabled by default. Depending on the actual edition, later versions of Windows starting from version 1709 (\"Fall Creators Update\") do not allow the installation of the SMBv1 client anymore. This causes hosts running Samba not to be listed in the Explorer's \"Network (Neighborhood)\" views. While there is no connectivity problem and Samba will still run fine, users might want to have their Samba hosts to be listed by Windows automatically. implements a Web Service Discovery host daemon. This enables (Samba) hosts, like your local NAS device, to be found by Web Service Discovery Clients like Windows. The default settings should work for most installations, all you need to do is start enable . If the default configuration (advertise itself as the machine hostname in group \"WORKGROUP\") should be all you need in most cases. If you need, you can change configuration options by passing additional arguments to wsdd by adding them in (see the manual page for wsdd for details). does the same thing, but is written in C instead of Python. By default, it will look for the and values in .","title":"Windows 1709 or up does not discover the samba server in Network view"},{"location":"smb/#ios-files-can-no-longer-copy-to-samba-share-on-arch-linux-beginning-with-ios-145","text":"Beginning with IOS 14.5 attempting to transfer from a device running IOS using the \"Files\" app to a samba share on Arch Linux will result in the error: The operation couldn't be completed Operation canceled To correct this problem, add add the following to the global section of your and restart . Comment optional: ## addition for IOS Files transfer-to server vfs object = fruit streams_xattr See https://apple.stackexchange.com/q/424681 Apple.Stackexchange.com - \"The operation couldn't be completed\"/\"Operation canceled\" error message when saving to a Samba share via Files app.","title":"IOS Files can no longer copy-to Samba share on Arch Linux beginning with IOS 14.5"},{"location":"smb/#slow-initial-connections-from-certain-clients-without-other-performance-problems","text":"Some SMB clients, such as Solid Explorer for Android, take significantly longer to connect to Samba if they fail to resolve the NetBIOS name. Enabling will greatly speed up initial connections if this is the case. Since this is a bug in the client software, please report such cases to the authors of conflicting software.","title":"Slow initial connections from certain clients without other performance problems"},{"location":"smb/#see-also","text":"Official website Samba: An Introduction Samba 3.2.x HOWTO and Reference Guide (outdated but still most extensive documentation) Wikipedia Gentoo:Samba/Guide Debian:Samba/ServerSimple KSMBD - A linux kernel server which implements SMB3 protocol in kernel space for sharing files over network. Category:Network sharing Category:Servers","title":"See also"},{"location":"smb2/","text":"Samba This document describes how to mount CIFS shares permanently. The shares might be hosted on a Windows computer/server, or on a Linux/UNIX server running Samba . This document also applies to SMBFS shares, which are similar to CIFS but are deprecated and should be avoided if possible ( link ). (This document does not describe how to host the shares yourself, only how to access shares that are hosted somewhere else. For hosting shares, use Samba .) Prerequisites We're assuming that: - Network connections have been configured properly. - Your local (Ubuntu) username is ubuntuusername . - Share username on Windows computer is msusername . - Share password on Windows computer is mspassword . - The Windows computer's name is servername this can be either an IP address or an assigned name). - The name of the share is sharename . - You want to mount the share in /media/windowsshare . CIFS installation On older systems: Mounting unprotected (guest) network folders First, let's create the mount directory. You will need a separate directory for each mount. Then edit your /etc/fstab file (with root privileges) to add this line: Where - guest indicates you don't need a password to access the share, - uid=1000 makes the Linux user specified by the id the owner of the mounted share, allowing them to rename files, - iocharset=utf8 allows access to files with names in non-English languages. This doesn't work with shares of devices like the Buffalo Tera Station, or Windows machines that export their shares using ISO8895-15. - If there is any space in the server path , you need to replace it by \\040, for example //servername/My\\040Documents` After you add the entry to /etc/fstab type: This will (re)mount all entries listed in /etc/fstab. Mount password protected network folders The quickest way to auto-mounting a password-protected share is to edit /etc/fstab (with root privileges), to add this line: This is not a good idea however: /etc/fstab is readable by everyone and so is your Windows password in it. The way around this is to use a credentials file. This is a file that contains just the username and password. Using a text editor, create a file for your remote servers logon credential: Enter your Windows username and password in the file: Save the file, exit the editor. Change the permissions of the file to prevent unwanted access to your credentials: Then edit your /etc/fstab file (with root privileges) to add this line (replacing the insecure line in the example above, if you added it): Save the file, exit the editor. Finally, test the fstab entry by issuing: If there are no errors, you should test how it works after a reboot. Your remote share should mount automatically. Special permissions If you need special permission (like chmod etc.), you'll need to add a uid (short for 'user id') or gid (for 'group id') parameter to the share's mount options. Mount password protected shares using libpam_mount (Ubuntu 9.04) In addition to the initial assumptions, we're assuming that - Your username and password are the same on the Ubuntu machine and on the network drive. Install libpam-mount: Edit /etc/security/pam_mount.conf.xml using your preferred text editor. First, we're moving the user specific config bits to a file which users can actually edit themselves: remove the commenting tags ( ) surrounding the section called . Save the file when done. With this in place, users can create their own \\~/.pam_mount.conf.xml. Add the following: Troubleshooting Login errors If you get the error \"mount error(13) permission denied\", then the server denied your access. Here are the first things to check: - Are you using a valid username and password? Does that account really have access to this folder? - Do you have whitespace in your credentials file? It should be , not . - Do you need a domain? For example, if you are told that your username is , then actually your username is and your domain is . The fstab entry should read: Or: - The security and version settings are interrelated. SMB1 is insecure and no longer supported by default. At first, try to not specify either security or version: do not specify or . If you still have authentication errors then you may need to specify either or or both. You can try the options listed at the [ mount.cifs man page ](http://manpages.ubuntu.com/manpages/raring/en/man8/mount.cifs.8.html \"wikilink\") . The man page list leaves out the option for some reason, but you should try that one as well (`[`see `discussion ](https://bugs.launchpad.net/ubuntu/+source/cifs-utils/+bug/1113395 \"wikilink\") ). Unprotected network folder won't automount I've had a situation where an unprotected network folder wouldn't automount during bootup, but after manually entering \"sudo mount -a\" was mounted correctly. I solved this by replacing the \"guest\" option by \"username=guest,password=\". If anyone has an explanation for this, please leave a comment. Mount during login instead of boot If for some reason/etc/rc0.d/S31umountnfs.sh (networking problems for example) the automatic mounting during boot doesn't work, you can add the \"noauto\" parameter to your smbfs fstab entry and then have the share mounted at login. In /etc/fstab: In /etc/rc.local: Slow shutdown due to a CIFS/Network Manager bug If you use Network Manager, and are getting really slow shutdowns, it's probably because NM shuts down before unmounting the network shares. That will cause CIFS to hang and wait for 60 seconds or so. Here's how to fix it:/etc/rc0.d/S31umountnfs.sh Ubuntu 12.04 already runs umountnfs.sh at reboot and shutdown by default (/etc/rc0.d/S31umountnfs.sh and /etc/rc6.d/S31umountnfs.sh) so this is no longer necessary. CIFS Options Deprecated 20 Feb 2008 TW Using dmask or fmask in the fstab file produces the following warnings: WARNING: CIFS mount option 'dmask' is deprecated. Use 'dir_mode' instead. WARNING: CIFS mount option 'fmask' is deprecated. Use 'file_mode' instead. Instead use this format: file_mode=0777,dir_mode=0777 . Or in some cases you might need to use file_mode=0777,dir_mode=0777,nounix ( see discussion ) \\== Use of tilde in pathnames such as \"credentials=\\~/.smbcredentials\" 20 Feb 2008 TW Curiously, using credentials=\\~/.smbcredentials in fstab didn't work. I had to use the full path, i.e. /home/username/.smbcredentials (This is likely because the tilde \"\\~\" is only a shell short-hand alias for \"$HOME\"; it isn't something recognized system-wide by all programs, especially not in a system file table where the concept of \"HOME\" doesn't really exist. -Ian!) CategoryDocumentation","title":"Samba_ubuntu"},{"location":"smb2/#samba","text":"This document describes how to mount CIFS shares permanently. The shares might be hosted on a Windows computer/server, or on a Linux/UNIX server running Samba . This document also applies to SMBFS shares, which are similar to CIFS but are deprecated and should be avoided if possible ( link ). (This document does not describe how to host the shares yourself, only how to access shares that are hosted somewhere else. For hosting shares, use Samba .)","title":"Samba"},{"location":"smb2/#prerequisites","text":"We're assuming that: - Network connections have been configured properly. - Your local (Ubuntu) username is ubuntuusername . - Share username on Windows computer is msusername . - Share password on Windows computer is mspassword . - The Windows computer's name is servername this can be either an IP address or an assigned name). - The name of the share is sharename . - You want to mount the share in /media/windowsshare .","title":"Prerequisites"},{"location":"smb2/#cifs-installation","text":"On older systems:","title":"CIFS installation"},{"location":"smb2/#mounting-unprotected-guest-network-folders","text":"First, let's create the mount directory. You will need a separate directory for each mount. Then edit your /etc/fstab file (with root privileges) to add this line: Where - guest indicates you don't need a password to access the share, - uid=1000 makes the Linux user specified by the id the owner of the mounted share, allowing them to rename files, - iocharset=utf8 allows access to files with names in non-English languages. This doesn't work with shares of devices like the Buffalo Tera Station, or Windows machines that export their shares using ISO8895-15. - If there is any space in the server path , you need to replace it by \\040, for example //servername/My\\040Documents` After you add the entry to /etc/fstab type: This will (re)mount all entries listed in /etc/fstab.","title":"Mounting unprotected (guest) network folders"},{"location":"smb2/#mount-password-protected-network-folders","text":"The quickest way to auto-mounting a password-protected share is to edit /etc/fstab (with root privileges), to add this line: This is not a good idea however: /etc/fstab is readable by everyone and so is your Windows password in it. The way around this is to use a credentials file. This is a file that contains just the username and password. Using a text editor, create a file for your remote servers logon credential: Enter your Windows username and password in the file: Save the file, exit the editor. Change the permissions of the file to prevent unwanted access to your credentials: Then edit your /etc/fstab file (with root privileges) to add this line (replacing the insecure line in the example above, if you added it): Save the file, exit the editor. Finally, test the fstab entry by issuing: If there are no errors, you should test how it works after a reboot. Your remote share should mount automatically.","title":"Mount password protected network folders"},{"location":"smb2/#special-permissions","text":"If you need special permission (like chmod etc.), you'll need to add a uid (short for 'user id') or gid (for 'group id') parameter to the share's mount options.","title":"Special permissions"},{"location":"smb2/#mount-password-protected-shares-using-libpam_mount-ubuntu-904","text":"In addition to the initial assumptions, we're assuming that - Your username and password are the same on the Ubuntu machine and on the network drive. Install libpam-mount: Edit /etc/security/pam_mount.conf.xml using your preferred text editor. First, we're moving the user specific config bits to a file which users can actually edit themselves: remove the commenting tags ( ) surrounding the section called . Save the file when done. With this in place, users can create their own \\~/.pam_mount.conf.xml. Add the following:","title":"Mount password protected shares using libpam_mount (Ubuntu 9.04)"},{"location":"smb2/#troubleshooting","text":"","title":"Troubleshooting"},{"location":"smb2/#login-errors","text":"If you get the error \"mount error(13) permission denied\", then the server denied your access. Here are the first things to check: - Are you using a valid username and password? Does that account really have access to this folder? - Do you have whitespace in your credentials file? It should be , not . - Do you need a domain? For example, if you are told that your username is , then actually your username is and your domain is . The fstab entry should read: Or: - The security and version settings are interrelated. SMB1 is insecure and no longer supported by default. At first, try to not specify either security or version: do not specify or . If you still have authentication errors then you may need to specify either or or both. You can try the options listed at the [ mount.cifs man page ](http://manpages.ubuntu.com/manpages/raring/en/man8/mount.cifs.8.html \"wikilink\") . The man page list leaves out the option for some reason, but you should try that one as well (`[`see `discussion ](https://bugs.launchpad.net/ubuntu/+source/cifs-utils/+bug/1113395 \"wikilink\") ).","title":"Login errors"},{"location":"smb2/#unprotected-network-folder-wont-automount","text":"I've had a situation where an unprotected network folder wouldn't automount during bootup, but after manually entering \"sudo mount -a\" was mounted correctly. I solved this by replacing the \"guest\" option by \"username=guest,password=\". If anyone has an explanation for this, please leave a comment.","title":"Unprotected network folder won't automount"},{"location":"smb2/#mount-during-login-instead-of-boot","text":"If for some reason/etc/rc0.d/S31umountnfs.sh (networking problems for example) the automatic mounting during boot doesn't work, you can add the \"noauto\" parameter to your smbfs fstab entry and then have the share mounted at login. In /etc/fstab: In /etc/rc.local:","title":"Mount during login instead of boot"},{"location":"smb2/#slow-shutdown-due-to-a-cifsnetwork-manager-bug","text":"If you use Network Manager, and are getting really slow shutdowns, it's probably because NM shuts down before unmounting the network shares. That will cause CIFS to hang and wait for 60 seconds or so. Here's how to fix it:/etc/rc0.d/S31umountnfs.sh Ubuntu 12.04 already runs umountnfs.sh at reboot and shutdown by default (/etc/rc0.d/S31umountnfs.sh and /etc/rc6.d/S31umountnfs.sh) so this is no longer necessary.","title":"Slow shutdown due to a CIFS/Network Manager bug"},{"location":"smb2/#cifs-options-deprecated","text":"20 Feb 2008 TW Using dmask or fmask in the fstab file produces the following warnings: WARNING: CIFS mount option 'dmask' is deprecated. Use 'dir_mode' instead. WARNING: CIFS mount option 'fmask' is deprecated. Use 'file_mode' instead. Instead use this format: file_mode=0777,dir_mode=0777 . Or in some cases you might need to use file_mode=0777,dir_mode=0777,nounix ( see discussion )","title":"CIFS Options Deprecated"},{"location":"smb2/#use-of-tilde-in-pathnames-such-as-credentialssmbcredentials","text":"20 Feb 2008 TW Curiously, using credentials=\\~/.smbcredentials in fstab didn't work. I had to use the full path, i.e. /home/username/.smbcredentials (This is likely because the tilde \"\\~\" is only a shell short-hand alias for \"$HOME\"; it isn't something recognized system-wide by all programs, especially not in a system file table where the concept of \"HOME\" doesn't really exist. -Ian!) CategoryDocumentation","title":"\\== Use of tilde in pathnames such as \"credentials=\\~/.smbcredentials\""},{"location":"softwarearchitecture/","text":"software architecture Software Architecture: The Hard Parts Modern Trade-Off Analyses for Distributed Architectures Introduction There are no easy decisions in software architecture. Instead, there are many hard parts -- difficult problems or issues with no best practices -- that force you to choose among various compromises. The best design an architect can create is the least worst collection of trade-offs -- no single architecture characteristic excels as it would alone, but the balance of all the competing architecture characteristics promote project success. The Importance of Data in Architecture It has been said that data is the most important asset in a company. Businesses want to extract value from the data that they have and are finding new ways to deploy data in decision making. Operational data Data that is used to run the business. This data is defined as Online Transactional Processing (OLTP) data. Involves Create, Read, Update, Delete (CRUD) operations. Analytical data Data that is used to analyze the business. This data is defined as Online Analytical Processing (OLAP) data. Involves Read operations. Architectural Decisions Records ADR : A short noun phrase containing the architecture decision Context In this section of the ADR we will add a short one- or two-sentence description of the problem, and list the alternative solutions. Decision In this section we will state the architecture decision and provide a detailed justification of the decision. Consequences In this section of the ADR we will describe any consequences after the decision is applied, and also discuss the trade-offs that were considered.","title":"softwarearchitecture"},{"location":"softwarearchitecture/#software-architecture","text":"Software Architecture: The Hard Parts Modern Trade-Off Analyses for Distributed Architectures","title":"software architecture"},{"location":"softwarearchitecture/#introduction","text":"There are no easy decisions in software architecture. Instead, there are many hard parts -- difficult problems or issues with no best practices -- that force you to choose among various compromises. The best design an architect can create is the least worst collection of trade-offs -- no single architecture characteristic excels as it would alone, but the balance of all the competing architecture characteristics promote project success.","title":"Introduction"},{"location":"softwarearchitecture/#the-importance-of-data-in-architecture","text":"It has been said that data is the most important asset in a company. Businesses want to extract value from the data that they have and are finding new ways to deploy data in decision making. Operational data Data that is used to run the business. This data is defined as Online Transactional Processing (OLTP) data. Involves Create, Read, Update, Delete (CRUD) operations. Analytical data Data that is used to analyze the business. This data is defined as Online Analytical Processing (OLAP) data. Involves Read operations.","title":"The Importance of Data in Architecture"},{"location":"softwarearchitecture/#architectural-decisions-records","text":"ADR : A short noun phrase containing the architecture decision Context In this section of the ADR we will add a short one- or two-sentence description of the problem, and list the alternative solutions. Decision In this section we will state the architecture decision and provide a detailed justification of the decision. Consequences In this section of the ADR we will describe any consequences after the decision is applied, and also discuss the trade-offs that were considered.","title":"Architectural Decisions Records"},{"location":"sprint/","text":"sprint s3 VegFru 25 classes S4 full datasets: time, results torch/tf custom data training: duration obj det coco yolox s5 doc","title":"sprint"},{"location":"sprint/#sprint","text":"","title":"sprint"},{"location":"sprint/#s3","text":"","title":"s3"},{"location":"sprint/#vegfru","text":"25 classes","title":"VegFru"},{"location":"sprint/#s4","text":"full datasets: time, results torch/tf custom data training: duration obj det coco yolox","title":"S4"},{"location":"sprint/#s5","text":"doc","title":"s5"},{"location":"ssh/","text":"ssh sudo pacman -S openssh sudo systemctl enable --now sshd If the username on your local machine matches the one on the server you are trying to connect to, you can just type: ssh host_ip_address And hit Enter. ssh your_username@host_ip_address To open SSH port on Manjaro, you can follow these steps 1: Make sure sshd (ssh daemon) is active and listening to incoming connections on the computer you are trying to access: systemctl status sshd If it\u2019s inactive, start it using: systemctl start sshd To automatically start it on boot: systemctl enable sshd Also allow incoming connections in your firewall. ssh-keygen -t rsa -b 2048 Generating public/private rsa key pair. Enter file in which to save the key (/home/username/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/username/.ssh/id_rsa. Your public key has been saved in /home/username/.ssh/id_rsa.pub. $ ssh-copy-id id@server id@server's password:","title":"ssh"},{"location":"ssh/#ssh","text":"sudo pacman -S openssh sudo systemctl enable --now sshd If the username on your local machine matches the one on the server you are trying to connect to, you can just type: ssh host_ip_address And hit Enter. ssh your_username@host_ip_address To open SSH port on Manjaro, you can follow these steps 1: Make sure sshd (ssh daemon) is active and listening to incoming connections on the computer you are trying to access: systemctl status sshd If it\u2019s inactive, start it using: systemctl start sshd To automatically start it on boot: systemctl enable sshd Also allow incoming connections in your firewall. ssh-keygen -t rsa -b 2048 Generating public/private rsa key pair. Enter file in which to save the key (/home/username/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/username/.ssh/id_rsa. Your public key has been saved in /home/username/.ssh/id_rsa.pub. $ ssh-copy-id id@server id@server's password:","title":"ssh"},{"location":"stable_diffusion/","text":"stable diffusion negative prompt \"longbody, lowres, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality\" Examples Source Input Output (no preprocessor) (no preprocessor)","title":"stable_diffussion"},{"location":"stable_diffusion/#stable-diffusion","text":"","title":"stable diffusion"},{"location":"stable_diffusion/#negative-prompt","text":"\"longbody, lowres, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality\"","title":"negative prompt"},{"location":"stable_diffusion/#examples","text":"Source Input Output (no preprocessor) (no preprocessor)","title":"Examples"},{"location":"sway/","text":"sway You may combine output commands into one, like so: output HDMI-A-1 mode 1920x1080 pos 1920 0 bg ~/wallpaper.png stretch You can get a list of output names with swaymsg -t get_outputs. You may also match any output by using the output name \"*\". Additionally, \"-\" can be used to match the focused output by name and \"--\" can be used to match the focused output by its identifier. Some outputs may have different names when disconnecting and reconnecting. To identify these, the name can be substituted for a string consisting of the make, model and serial which you can get from swaymsg -t get_outputs. Each value must be separated by one space. For example: output \"Some Company ABC123 0x00000000\" pos 1920 0 swaymsg -t get_outputs output mode|resolution|res [--custom] [@ [Hz]] Configures the specified output to use the given mode. Modes are a combination of width and height (in pixels) and a refresh rate that your display can be configured to use. For a list of available modes for each output, use swaymsg -t get_outputs. To set a custom mode not listed in the list of available modes, use --custom. You should probably only use this if you know what you're doing. Examples: output HDMI-A-1 mode 1920x1080 output HDMI-A-1 mode 1920x1080@60Hz output position|pos Places the specified output at the specific position in the global coordinate space. The cursor may only be moved between immediately adjacent outputs. If scaling is active, it has to be considered when positioning. For example, if the scaling factor for the left output is 2, the relative position for the right output has to be divided by 2. The reference point is the top left corner so if you want the bottoms aligned this has to be considered as well. Example: output HDMI1 scale 2 output HDMI1 pos 0 1020 res 3200x1800 output eDP1 pos 1600 0 res 1920x1080 Note that the left x-pos of eDP1 is 1600 = 3200/2 and the bottom y-pos is 1020 + (1800 / 2) = 1920 = 0 + 1920 output scale Scales the specified output by the specified scale factor. An integer is recommended, but fractional values are also supported. If a fractional value are specified, be warned that it is not possible to faithfully represent the contents of your windows - they will be rendered at the next highest integer scale factor and downscaled. You may be better served by setting an integer scale factor and adjusting the font size of your applications to taste. HiDPI isn't supported with Xwayland clients (windows will blur). output scale_filter linear|nearest|smart Indicates how to scale application buffers that are rendered at a scale lower than the output's configured scale, such as lo-dpi applications on hi-dpi screens. Linear is smoother and blurrier, nearest (also known as nearest neighbor) is sharper and blockier. Setting \"smart\" will apply nearest scaling when the output has an integer scale factor, otherwise linear. The default is \"smart\". output subpixel rgb|bgr|vrgb|vbgr|none Manually sets the subpixel hinting for the specified output. This value is usually auto-detected, but some displays may misreport their subpixel geometry. Using the correct subpixel hinting allows for sharper text. Incorrect values will result in blurrier text. When changing this via swaymsg, some applications may need to be restarted to use the new value. output background|bg [ ] Sets the wallpaper for the given output to the specified file, using the given scaling mode (one of \"stretch\", \"fill\", \"fit\", \"center\", \"tile\"). If the specified file cannot be accessed or if the image does fill the entire output, a fallback color may be provided to cover the rest of the output. fallback_color should be specified as #RRGGBB. Alpha is not supported. output background|bg solid_color Sets the background of the given output to the specified color. color should be specified as #RRGGBB. Alpha is not supported. output transform [clockwise|anticlockwise] Sets the background transform to the given value. Can be one of \"90\", \"180\", \"270\" for rotation; or \"flipped\", \"flipped-90\", \"flipped-180\", \"flipped-270\" to apply a rotation and flip, or \"normal\" to apply no transform. If a single output is chosen and a rotation direction is specified (clockwise or anticlockwise) then the transform is added or subtracted from the current transform. output disable|enable Enables or disables the specified output (all outputs are enabled by default). output toggle Toggle the specified output. output dpms on|off Enables or disables the specified output via DPMS. To turn an output off (ie. blank the screen but keep workspaces as-is), one can set DPMS to off. output max_render_time off| When set to a positive number of milliseconds, enables delaying output rendering to reduce latency. The rendering is delayed in such a way as to leave the specified number of milliseconds before the next presentation for rendering. The output rendering normally takes place immediately after a presentation (vblank, buffer flip, etc.) and the frame callbacks are sent to surfaces immediately after the rendering to give surfaces the most time to draw their next frame. This results in slightly below 2 frames of latency between the surface rendering and committing new contents, and the contents being shown on screen, on average. When the output rendering is delayed, the frame callbacks are sent immediately after presentation, and the surfaces have a small timespan (1 / (refresh rate) - max_render_time) to render and commit new contents to be shown on the next presentation, resulting in below 1 frame of latency. To set this up for optimal latency: 1. Launch some full-screen application that renders continuously, like glxgears. 2. Start with max_render_time 1. Increment by 1 if you see frame drops. To achieve even lower latency, see the max_render_time surface property in sway(5). Note that this property has an effect only on backends which report the presentation timestamp and the predicted output refresh rate\u2014the DRM and the Wayland backends. Furthermore, under the Wayland backend the optimal max_render_time value may vary based on the parent compositor rendering timings. .config/sway/config.d/output.conf output DP-1 resolution 3440x1440@144Hz position 0,0 adaptive_sync off output DP-4 resolution 2560x1440@59.951Hz transform 270 position 3440,0 .config/sway/definitions.d/custom.conf set $idle_timeout 240 set $locking_timeout 1200 set $screen_timeout 1200 set $swayidle swayidle -w \\ timeout $idle_timeout 'light -G > /tmp/brightness && light -S 10' resume 'light -S $([ -f /tmp/brightness ] && cat /tmp/brightness || echo 100%)' \\ timeout $locking_timeout 'exec $locking' \\ timeout $screen_timeout 'swaymsg \"output * dpms off\"' \\ resume 'swaymsg \"output * dpms on\"' \\ before-sleep 'playerctl pause' \\ before-sleep 'exec $locking' .config/waybar/config.jsonc // ============================================================================= // // Waybar configuration // // Configuration reference: https://github.com/Alexays/Waybar/wiki/Configuration // // ============================================================================= { \"include\": [ \"/usr/share/sway/templates/waybar/config.jsonc\" ], // ------------------------------------------------------------------------- // Global configuration // ------------------------------------------------------------------------- \"layer\": \"top\", // If height property would be not present, it'd be calculated dynamically \"height\": 30, \"position\": \"top\", \"modules-left\": [\"custom/menu\", \"sway/workspaces\", \"custom/scratchpad\"], \"modules-center\": [\"custom/wf-recorder\", \"sway/mode\", \"custom/weather\"], \"modules-right\": [ // informational \"sway/language\", \"custom/github\", \"custom/clipboard\", \"custom/zeit\", \"cpu\", \"memory\", \"battery\", \"temperature\", // connecting \"network\", \"bluetooth\", // media \"custom/playerctl\", \"idle_inhibitor\", \"custom/dnd\", \"pulseaudio\", \"backlight\", // system \"custom/adaptive-light\", \"custom/sunset\", \"custom/pacman\", \"tray\", \"clock\" ], // ------------------------------------------------------------------------- // Modules // ------------------------------------------------------------------------- \"battery\": { \"interval\": 30, \"states\": { \"warning\": 30, \"critical\": 15 }, \"format-charging\": \"\uf583 {capacity}%\", \"format\": \"{icon} {capacity}%\", \"format-icons\": [\"\uf582\", \"\uf579\", \"\uf57a\", \"\uf57d\", \"\uf57f\", \"\uf578\"], \"tooltip\": true }, \"clock\": { \"interval\": 60, \"format\": \"{:%e %b %Y %H:%M}\", \"tooltip\": true, \"tooltip-format\": \"{:%B %Y}\\n{calendar}\", \"on-click\": \"swaymsg exec \\\\$calendar\" }, \"cpu\": { \"interval\": 5, \"format\": \"\ufb19 {usage}%\", \"states\": { \"warning\": 70, \"critical\": 90 }, \"on-click\": \"swaymsg exec \\\\$term_float htop\" }, \"memory\": { \"interval\": 5, \"format\": \"\uf85a {}%\", \"states\": { \"warning\": 70, \"critical\": 90 }, \"on-click\": \"swaymsg exec \\\\$term_float htop\" }, \"network\": { \"interval\": 5, \"format-wifi\": \"\uf1eb \", \"format-ethernet\": \"\uf6ff\", \"format-disconnected\": \"\ufaa9\", \"tooltip-format\": \"{ifname} ({essid}): {ipaddr}\", \"on-click\": \"swaymsg exec \\\\$term_float nmtui\" }, \"sway/mode\": { \"format\": \"{}\", \"tooltip\": false }, \"idle_inhibitor\": { \"format\": \"{icon}\", \"format-icons\": { \"activated\": \"\uf9b2\", \"deactivated\": \"\uf9b1\" }, \"tooltip\": true, \"tooltip-format-activated\": \"power-saving disabled\", \"tooltip-format-deactivated\": \"power-saving enabled\" }, \"backlight\": { \"format\": \"{icon} {percent}%\", \"format-icons\": [\"\uf5dd\", \"\uf5de\", \"\uf5df\"], \"on-scroll-up\": \"swaymsg exec \\\\$brightness_up\", \"on-scroll-down\": \"swaymsg exec \\\\$brightness_down\" }, \"pulseaudio\": { \"scroll-step\": 5, \"format\": \"{icon} {volume}%{format_source}\", \"format-muted\": \"\ufa80 {format_source}\", \"format-source\": \"\", \"format-source-muted\": \" \uf86c\", \"format-icons\": { \"headphone\": \"\uf7ca\", \"headset\": \"\uf7cd\", \"default\": [\"\ufa7e\", \"\ufa7f\", \"\ufa7d\"] }, \"tooltip-format\": \"{icon}\u200a{volume}% {format_source}\", \"on-click\": \"swaymsg exec \\\\$pulseaudio\", \"on-click-middle\": \"swaymsg exec \\\\$volume_mute\", \"on-scroll-up\": \"swaymsg exec \\\\$volume_up\", \"on-scroll-down\": \"swaymsg exec \\\\$volume_down\" }, \"temperature\": { \"critical-threshold\": 90, \"interval\": 5, \"format\": \"{icon} {temperatureC}\u00b0\", \"format-icons\": [\"\uf2cb\", \"\uf2c9\", \"\uf2c8\"], \"tooltip\": false, \"on-click\": \"swaymsg exec \\\"\\\\$term_float watch sensors\\\"\" }, \"tray\": { \"icon-size\": 21, \"spacing\": 5 }, \"custom/pacman\": { \"format\": \"\uf53b {}\", \"interval\": 3600, \"exec-if\": \"[ $(pamac checkupdates -q | wc -l) -gt 0 ]\", \"exec\": \"pamac checkupdates -q | wc -l\", \"on-click\": \"pamac-manager --updates; pkill -RTMIN+4 waybar\", \"signal\": 4 }, \"custom/menu\": { \"format\": \"\uf312\", \"on-click\": \"swaymsg exec \\\\$menu\", \"tooltip\": false }, \"bluetooth\": { \"format\": \"\uf5ae\", \"format-disabled\": \"\uf5b1\", \"on-click\": \"swaymsg exec \\\\$bluetooth\", \"on-click-right\": \"rfkill toggle bluetooth\", \"tooltip-format\": \"{}\" }, \"sway/language\": { \"format\": \"\uf11c {}\", \"min-length\": 5, \"tooltip\": false, \"on-click\": \"swaymsg input $(swaymsg -t get_inputs --raw | jq '[.[] | select(.type == \\\"keyboard\\\")][0] | .identifier') xkb_switch_layout next\" }, \"custom/scratchpad\": { \"interval\": \"once\", \"return-type\": \"json\", \"format\": \"{icon}\", \"format-icons\": { \"one\": \"\ufaae\", \"many\": \"\ufab1\" }, \"exec\": \"/bin/sh /usr/share/sway/scripts/scratchpad.sh\", \"on-click\": \"swaymsg 'scratchpad show'\", \"signal\": 7 }, \"custom/sunset\": { \"interval\": \"once\", \"tooltip\": true, \"return-type\": \"json\", \"format\": \"{icon}\", \"format-icons\": { \"on\": \"\uf834\", \"off\": \"\uf400\" }, \"exec\": \"fallback_latitude=50.1 fallback_longitude=8.7 latitude= longitude= /usr/share/sway/scripts/sunset.sh\", \"on-click\": \"/usr/share/sway/scripts/sunset.sh toggle; pkill -RTMIN+6 waybar\", \"exec-if\": \"/usr/share/sway/scripts/sunset.sh check\", \"signal\": 6 }, \"custom/wf-recorder\": { \"interval\": \"once\", \"return-type\": \"json\", \"format\": \"{}\", \"tooltip-format\": \"{tooltip}\", \"exec\": \"echo '{\\\"class\\\": \\\"recording\\\",\\\"text\\\":\\\"\uf949\\\",\\\"tooltip\\\":\\\"press $mod+Esc to stop recording\\\"}'\", \"exec-if\": \"pgrep wf-recorder\", \"on-click\": \"killall -s SIGINT wf-recorder\", \"signal\": 8 }, \"custom/github\": { \"interval\": 300, \"tooltip\": false, \"return-type\": \"json\", \"format\": \"\uf408 {}\", \"exec\": \"gh api '/notifications' -q '{ text: length }' | cat -\", \"exec-if\": \"[ -x \\\"$(command -v gh)\\\" ] && gh auth status 2>&1 | grep -q -m 1 'Logged in' && gh api '/notifications' -q 'length' | grep -q -m 1 '0' ; test $? -eq 1\", \"on-click\": \"xdg-open https://github.com/notifications && sleep 30 && pkill -RTMIN+4 waybar\", \"signal\": 4 }, \"custom/playerctl\": { \"interval\": \"once\", \"tooltip\": true, \"return-type\": \"json\", \"format\": \"{icon}\", \"format-icons\": { \"Playing\": \"\uf8e5\", \"Paused\": \"\uf90c\" }, \"exec\": \"playerctl metadata --format '{\\\"alt\\\": \\\"{{status}}\\\", \\\"tooltip\\\": \\\"{{playerName}}: {{markup_escape(title)}} - {{markup_escape(artist)}}\\\" }'\", \"on-click\": \"playerctl play-pause; pkill -RTMIN+5 waybar\", \"on-click-right\": \"playerctl next; pkill -RTMIN+5 waybar\", \"on-scroll-up\": \"playerctl position 10+; pkill -RTMIN+5 waybar\", \"on-scroll-down\": \"playerctl position 10-; pkill -RTMIN+5 waybar\", \"signal\": 5 }, \"custom/clipboard\": { \"format\": \"\uf691\", \"interval\": \"once\", \"return-type\": \"json\", \"on-click\": \"swaymsg -q exec '$clipboard'; pkill -RTMIN+9 waybar\", \"on-click-right\": \"swaymsg -q exec '$clipboard-del'; pkill -RTMIN+9 waybar\", \"on-click-middle\": \"rm -f ~/.cache/cliphist/db; pkill -RTMIN+9 waybar\", \"exec\": \"printf '{\\\"tooltip\\\":\\\"%s\\\"}' $(cliphist list | wc -l)' item(s) in the clipboard\\r(Mid click to clear)'\", \"exec-if\": \"[ -x \\\"$(command -v cliphist)\\\" ] && [ $(cliphist list | wc -l) -gt 0 ]\", \"signal\": 9 }, \"custom/weather\": { \"icon-size\": 42, \"format\": \"{icon} {}\", \"tooltip\": true, \"interval\": 3600, // accepts -c/--city -t/--temperature -d/--distance \"exec\": \"/usr/share/sway/scripts/weather.py -c sg\", \"return-type\": \"json\", \"format-icons\": { \"Unknown\": \"\ue370\", \"Cloudy\": \"\ufa8f\", \"Fog\": \"\ue313\", \"HeavyRain\": \"\ue318\", \"HeavyShowers\": \"\ue319\", \"HeavySnow\": \"\ue35e\", \"HeavySnowShowers\": \"\ufc15\", \"LightRain\": \"\ue306\", \"LightShowers\": \"\ue309\", \"LightSleet\": \"\ue3ad\", \"LightSleetShowers\": \"\ue31a\", \"LightSnow\": \"\ue31a\", \"LightSnowShowers\": \"\ufb7d\", \"PartlyCloudy\": \"\ue30c\", \"Sunny\": \"\ue30d\", \"ThunderyHeavyRain\": \"\ufb7c\", \"ThunderyShowers\": \"\ue30e\", \"ThunderySnowShowers\": \"\ue366\", \"VeryCloudy\": \"\ue33d\" } }, \"custom/zeit\": { \"return-type\": \"json\", \"interval\": \"once\", \"format\": \"{icon}\", \"format-icons\": { \"tracking\": \"\ufab4\", \"stopped\": \"\uf236\" }, \"exec\": \"/bin/sh /usr/share/sway/scripts/zeit.sh status\", \"on-click\": \"/bin/sh /usr/share/sway/scripts/zeit.sh click; pkill -RTMIN+10 waybar\", \"exec-if\": \"[ -x \\\"$(command -v zeit)\\\" ]\", \"signal\": 10 }, \"custom/dnd\": { \"interval\": \"once\", \"return-type\": \"json\", \"format\": \"{}{icon}\", \"format-icons\": { \"default\": \"\uf868\", \"dnd\": \"\ufba1\" }, \"on-click\": \"makoctl mode | grep 'do-not-disturb' && makoctl mode -r do-not-disturb || makoctl mode -a do-not-disturb; pkill -RTMIN+11 waybar\", \"on-click-right\": \"makoctl restore\", \"exec\": \"printf '{\\\"alt\\\":\\\"%s\\\",\\\"tooltip\\\":\\\"mode: %s\\\"}' $(makoctl mode | grep -q 'do-not-disturb' && echo dnd || echo default) $(makoctl mode | tail -1)\", \"signal\": 11 }, \"custom/adaptive-light\": { \"interval\": \"once\", \"tooltip\": true, \"return-type\": \"json\", \"format\": \"{icon}\", \"format-icons\": { \"on\": \"\uf5e0\", \"off\": \"\uf5df\" }, \"exec\": \"/usr/share/sway/scripts/wluma.sh\", \"on-click\": \"/usr/share/sway/scripts/wluma.sh toggle; pkill -RTMIN+12 waybar\", \"exec-if\": \"/usr/share/sway/scripts/wluma.sh check\", \"signal\": 12 } }","title":"sway"},{"location":"sway/#sway","text":"You may combine output commands into one, like so: output HDMI-A-1 mode 1920x1080 pos 1920 0 bg ~/wallpaper.png stretch You can get a list of output names with swaymsg -t get_outputs. You may also match any output by using the output name \"*\". Additionally, \"-\" can be used to match the focused output by name and \"--\" can be used to match the focused output by its identifier. Some outputs may have different names when disconnecting and reconnecting. To identify these, the name can be substituted for a string consisting of the make, model and serial which you can get from swaymsg -t get_outputs. Each value must be separated by one space. For example: output \"Some Company ABC123 0x00000000\" pos 1920 0 swaymsg -t get_outputs","title":"sway"},{"location":"sway/#output-moderesolutionres-custom-hz","text":"Configures the specified output to use the given mode. Modes are a combination of width and height (in pixels) and a refresh rate that your display can be configured to use. For a list of available modes for each output, use swaymsg -t get_outputs. To set a custom mode not listed in the list of available modes, use --custom. You should probably only use this if you know what you're doing. Examples: output HDMI-A-1 mode 1920x1080 output HDMI-A-1 mode 1920x1080@60Hz","title":"output mode|resolution|res [--custom] [@[Hz]]"},{"location":"sway/#output-positionpos","text":"Places the specified output at the specific position in the global coordinate space. The cursor may only be moved between immediately adjacent outputs. If scaling is active, it has to be considered when positioning. For example, if the scaling factor for the left output is 2, the relative position for the right output has to be divided by 2. The reference point is the top left corner so if you want the bottoms aligned this has to be considered as well. Example: output HDMI1 scale 2 output HDMI1 pos 0 1020 res 3200x1800 output eDP1 pos 1600 0 res 1920x1080 Note that the left x-pos of eDP1 is 1600 = 3200/2 and the bottom y-pos is 1020 + (1800 / 2) = 1920 = 0 + 1920","title":"output position|pos"},{"location":"sway/#output-scale","text":"Scales the specified output by the specified scale factor. An integer is recommended, but fractional values are also supported. If a fractional value are specified, be warned that it is not possible to faithfully represent the contents of your windows - they will be rendered at the next highest integer scale factor and downscaled. You may be better served by setting an integer scale factor and adjusting the font size of your applications to taste. HiDPI isn't supported with Xwayland clients (windows will blur).","title":"output scale"},{"location":"sway/#output-scale_filter-linearnearestsmart","text":"Indicates how to scale application buffers that are rendered at a scale lower than the output's configured scale, such as lo-dpi applications on hi-dpi screens. Linear is smoother and blurrier, nearest (also known as nearest neighbor) is sharper and blockier. Setting \"smart\" will apply nearest scaling when the output has an integer scale factor, otherwise linear. The default is \"smart\".","title":"output scale_filter linear|nearest|smart"},{"location":"sway/#output-subpixel-rgbbgrvrgbvbgrnone","text":"Manually sets the subpixel hinting for the specified output. This value is usually auto-detected, but some displays may misreport their subpixel geometry. Using the correct subpixel hinting allows for sharper text. Incorrect values will result in blurrier text. When changing this via swaymsg, some applications may need to be restarted to use the new value.","title":"output subpixel rgb|bgr|vrgb|vbgr|none"},{"location":"sway/#output-backgroundbg","text":"Sets the wallpaper for the given output to the specified file, using the given scaling mode (one of \"stretch\", \"fill\", \"fit\", \"center\", \"tile\"). If the specified file cannot be accessed or if the image does fill the entire output, a fallback color may be provided to cover the rest of the output. fallback_color should be specified as #RRGGBB. Alpha is not supported.","title":"output background|bg []"},{"location":"sway/#output-backgroundbg-solid_color","text":"Sets the background of the given output to the specified color. color should be specified as #RRGGBB. Alpha is not supported.","title":"output background|bg solid_color"},{"location":"sway/#output-transform-clockwiseanticlockwise","text":"Sets the background transform to the given value. Can be one of \"90\", \"180\", \"270\" for rotation; or \"flipped\", \"flipped-90\", \"flipped-180\", \"flipped-270\" to apply a rotation and flip, or \"normal\" to apply no transform. If a single output is chosen and a rotation direction is specified (clockwise or anticlockwise) then the transform is added or subtracted from the current transform.","title":"output transform [clockwise|anticlockwise]"},{"location":"sway/#output-disableenable","text":"Enables or disables the specified output (all outputs are enabled by default).","title":"output disable|enable"},{"location":"sway/#output-toggle","text":"Toggle the specified output.","title":"output toggle"},{"location":"sway/#output-dpms-onoff","text":"Enables or disables the specified output via DPMS. To turn an output off (ie. blank the screen but keep workspaces as-is), one can set DPMS to off.","title":"output dpms on|off"},{"location":"sway/#output-max_render_time-off","text":"When set to a positive number of milliseconds, enables delaying output rendering to reduce latency. The rendering is delayed in such a way as to leave the specified number of milliseconds before the next presentation for rendering. The output rendering normally takes place immediately after a presentation (vblank, buffer flip, etc.) and the frame callbacks are sent to surfaces immediately after the rendering to give surfaces the most time to draw their next frame. This results in slightly below 2 frames of latency between the surface rendering and committing new contents, and the contents being shown on screen, on average. When the output rendering is delayed, the frame callbacks are sent immediately after presentation, and the surfaces have a small timespan (1 / (refresh rate) - max_render_time) to render and commit new contents to be shown on the next presentation, resulting in below 1 frame of latency. To set this up for optimal latency: 1. Launch some full-screen application that renders continuously, like glxgears. 2. Start with max_render_time 1. Increment by 1 if you see frame drops. To achieve even lower latency, see the max_render_time surface property in sway(5). Note that this property has an effect only on backends which report the presentation timestamp and the predicted output refresh rate\u2014the DRM and the Wayland backends. Furthermore, under the Wayland backend the optimal max_render_time value may vary based on the parent compositor rendering timings. .config/sway/config.d/output.conf output DP-1 resolution 3440x1440@144Hz position 0,0 adaptive_sync off output DP-4 resolution 2560x1440@59.951Hz transform 270 position 3440,0 .config/sway/definitions.d/custom.conf set $idle_timeout 240 set $locking_timeout 1200 set $screen_timeout 1200 set $swayidle swayidle -w \\ timeout $idle_timeout 'light -G > /tmp/brightness && light -S 10' resume 'light -S $([ -f /tmp/brightness ] && cat /tmp/brightness || echo 100%)' \\ timeout $locking_timeout 'exec $locking' \\ timeout $screen_timeout 'swaymsg \"output * dpms off\"' \\ resume 'swaymsg \"output * dpms on\"' \\ before-sleep 'playerctl pause' \\ before-sleep 'exec $locking' .config/waybar/config.jsonc // ============================================================================= // // Waybar configuration // // Configuration reference: https://github.com/Alexays/Waybar/wiki/Configuration // // ============================================================================= { \"include\": [ \"/usr/share/sway/templates/waybar/config.jsonc\" ], // ------------------------------------------------------------------------- // Global configuration // ------------------------------------------------------------------------- \"layer\": \"top\", // If height property would be not present, it'd be calculated dynamically \"height\": 30, \"position\": \"top\", \"modules-left\": [\"custom/menu\", \"sway/workspaces\", \"custom/scratchpad\"], \"modules-center\": [\"custom/wf-recorder\", \"sway/mode\", \"custom/weather\"], \"modules-right\": [ // informational \"sway/language\", \"custom/github\", \"custom/clipboard\", \"custom/zeit\", \"cpu\", \"memory\", \"battery\", \"temperature\", // connecting \"network\", \"bluetooth\", // media \"custom/playerctl\", \"idle_inhibitor\", \"custom/dnd\", \"pulseaudio\", \"backlight\", // system \"custom/adaptive-light\", \"custom/sunset\", \"custom/pacman\", \"tray\", \"clock\" ], // ------------------------------------------------------------------------- // Modules // ------------------------------------------------------------------------- \"battery\": { \"interval\": 30, \"states\": { \"warning\": 30, \"critical\": 15 }, \"format-charging\": \"\uf583 {capacity}%\", \"format\": \"{icon} {capacity}%\", \"format-icons\": [\"\uf582\", \"\uf579\", \"\uf57a\", \"\uf57d\", \"\uf57f\", \"\uf578\"], \"tooltip\": true }, \"clock\": { \"interval\": 60, \"format\": \"{:%e %b %Y %H:%M}\", \"tooltip\": true, \"tooltip-format\": \"{:%B %Y}\\n{calendar}\", \"on-click\": \"swaymsg exec \\\\$calendar\" }, \"cpu\": { \"interval\": 5, \"format\": \"\ufb19 {usage}%\", \"states\": { \"warning\": 70, \"critical\": 90 }, \"on-click\": \"swaymsg exec \\\\$term_float htop\" }, \"memory\": { \"interval\": 5, \"format\": \"\uf85a {}%\", \"states\": { \"warning\": 70, \"critical\": 90 }, \"on-click\": \"swaymsg exec \\\\$term_float htop\" }, \"network\": { \"interval\": 5, \"format-wifi\": \"\uf1eb \", \"format-ethernet\": \"\uf6ff\", \"format-disconnected\": \"\ufaa9\", \"tooltip-format\": \"{ifname} ({essid}): {ipaddr}\", \"on-click\": \"swaymsg exec \\\\$term_float nmtui\" }, \"sway/mode\": { \"format\": \"{}\", \"tooltip\": false }, \"idle_inhibitor\": { \"format\": \"{icon}\", \"format-icons\": { \"activated\": \"\uf9b2\", \"deactivated\": \"\uf9b1\" }, \"tooltip\": true, \"tooltip-format-activated\": \"power-saving disabled\", \"tooltip-format-deactivated\": \"power-saving enabled\" }, \"backlight\": { \"format\": \"{icon} {percent}%\", \"format-icons\": [\"\uf5dd\", \"\uf5de\", \"\uf5df\"], \"on-scroll-up\": \"swaymsg exec \\\\$brightness_up\", \"on-scroll-down\": \"swaymsg exec \\\\$brightness_down\" }, \"pulseaudio\": { \"scroll-step\": 5, \"format\": \"{icon} {volume}%{format_source}\", \"format-muted\": \"\ufa80 {format_source}\", \"format-source\": \"\", \"format-source-muted\": \" \uf86c\", \"format-icons\": { \"headphone\": \"\uf7ca\", \"headset\": \"\uf7cd\", \"default\": [\"\ufa7e\", \"\ufa7f\", \"\ufa7d\"] }, \"tooltip-format\": \"{icon}\u200a{volume}% {format_source}\", \"on-click\": \"swaymsg exec \\\\$pulseaudio\", \"on-click-middle\": \"swaymsg exec \\\\$volume_mute\", \"on-scroll-up\": \"swaymsg exec \\\\$volume_up\", \"on-scroll-down\": \"swaymsg exec \\\\$volume_down\" }, \"temperature\": { \"critical-threshold\": 90, \"interval\": 5, \"format\": \"{icon} {temperatureC}\u00b0\", \"format-icons\": [\"\uf2cb\", \"\uf2c9\", \"\uf2c8\"], \"tooltip\": false, \"on-click\": \"swaymsg exec \\\"\\\\$term_float watch sensors\\\"\" }, \"tray\": { \"icon-size\": 21, \"spacing\": 5 }, \"custom/pacman\": { \"format\": \"\uf53b {}\", \"interval\": 3600, \"exec-if\": \"[ $(pamac checkupdates -q | wc -l) -gt 0 ]\", \"exec\": \"pamac checkupdates -q | wc -l\", \"on-click\": \"pamac-manager --updates; pkill -RTMIN+4 waybar\", \"signal\": 4 }, \"custom/menu\": { \"format\": \"\uf312\", \"on-click\": \"swaymsg exec \\\\$menu\", \"tooltip\": false }, \"bluetooth\": { \"format\": \"\uf5ae\", \"format-disabled\": \"\uf5b1\", \"on-click\": \"swaymsg exec \\\\$bluetooth\", \"on-click-right\": \"rfkill toggle bluetooth\", \"tooltip-format\": \"{}\" }, \"sway/language\": { \"format\": \"\uf11c {}\", \"min-length\": 5, \"tooltip\": false, \"on-click\": \"swaymsg input $(swaymsg -t get_inputs --raw | jq '[.[] | select(.type == \\\"keyboard\\\")][0] | .identifier') xkb_switch_layout next\" }, \"custom/scratchpad\": { \"interval\": \"once\", \"return-type\": \"json\", \"format\": \"{icon}\", \"format-icons\": { \"one\": \"\ufaae\", \"many\": \"\ufab1\" }, \"exec\": \"/bin/sh /usr/share/sway/scripts/scratchpad.sh\", \"on-click\": \"swaymsg 'scratchpad show'\", \"signal\": 7 }, \"custom/sunset\": { \"interval\": \"once\", \"tooltip\": true, \"return-type\": \"json\", \"format\": \"{icon}\", \"format-icons\": { \"on\": \"\uf834\", \"off\": \"\uf400\" }, \"exec\": \"fallback_latitude=50.1 fallback_longitude=8.7 latitude= longitude= /usr/share/sway/scripts/sunset.sh\", \"on-click\": \"/usr/share/sway/scripts/sunset.sh toggle; pkill -RTMIN+6 waybar\", \"exec-if\": \"/usr/share/sway/scripts/sunset.sh check\", \"signal\": 6 }, \"custom/wf-recorder\": { \"interval\": \"once\", \"return-type\": \"json\", \"format\": \"{}\", \"tooltip-format\": \"{tooltip}\", \"exec\": \"echo '{\\\"class\\\": \\\"recording\\\",\\\"text\\\":\\\"\uf949\\\",\\\"tooltip\\\":\\\"press $mod+Esc to stop recording\\\"}'\", \"exec-if\": \"pgrep wf-recorder\", \"on-click\": \"killall -s SIGINT wf-recorder\", \"signal\": 8 }, \"custom/github\": { \"interval\": 300, \"tooltip\": false, \"return-type\": \"json\", \"format\": \"\uf408 {}\", \"exec\": \"gh api '/notifications' -q '{ text: length }' | cat -\", \"exec-if\": \"[ -x \\\"$(command -v gh)\\\" ] && gh auth status 2>&1 | grep -q -m 1 'Logged in' && gh api '/notifications' -q 'length' | grep -q -m 1 '0' ; test $? -eq 1\", \"on-click\": \"xdg-open https://github.com/notifications && sleep 30 && pkill -RTMIN+4 waybar\", \"signal\": 4 }, \"custom/playerctl\": { \"interval\": \"once\", \"tooltip\": true, \"return-type\": \"json\", \"format\": \"{icon}\", \"format-icons\": { \"Playing\": \"\uf8e5\", \"Paused\": \"\uf90c\" }, \"exec\": \"playerctl metadata --format '{\\\"alt\\\": \\\"{{status}}\\\", \\\"tooltip\\\": \\\"{{playerName}}: {{markup_escape(title)}} - {{markup_escape(artist)}}\\\" }'\", \"on-click\": \"playerctl play-pause; pkill -RTMIN+5 waybar\", \"on-click-right\": \"playerctl next; pkill -RTMIN+5 waybar\", \"on-scroll-up\": \"playerctl position 10+; pkill -RTMIN+5 waybar\", \"on-scroll-down\": \"playerctl position 10-; pkill -RTMIN+5 waybar\", \"signal\": 5 }, \"custom/clipboard\": { \"format\": \"\uf691\", \"interval\": \"once\", \"return-type\": \"json\", \"on-click\": \"swaymsg -q exec '$clipboard'; pkill -RTMIN+9 waybar\", \"on-click-right\": \"swaymsg -q exec '$clipboard-del'; pkill -RTMIN+9 waybar\", \"on-click-middle\": \"rm -f ~/.cache/cliphist/db; pkill -RTMIN+9 waybar\", \"exec\": \"printf '{\\\"tooltip\\\":\\\"%s\\\"}' $(cliphist list | wc -l)' item(s) in the clipboard\\r(Mid click to clear)'\", \"exec-if\": \"[ -x \\\"$(command -v cliphist)\\\" ] && [ $(cliphist list | wc -l) -gt 0 ]\", \"signal\": 9 }, \"custom/weather\": { \"icon-size\": 42, \"format\": \"{icon} {}\", \"tooltip\": true, \"interval\": 3600, // accepts -c/--city -t/--temperature -d/--distance \"exec\": \"/usr/share/sway/scripts/weather.py -c sg\", \"return-type\": \"json\", \"format-icons\": { \"Unknown\": \"\ue370\", \"Cloudy\": \"\ufa8f\", \"Fog\": \"\ue313\", \"HeavyRain\": \"\ue318\", \"HeavyShowers\": \"\ue319\", \"HeavySnow\": \"\ue35e\", \"HeavySnowShowers\": \"\ufc15\", \"LightRain\": \"\ue306\", \"LightShowers\": \"\ue309\", \"LightSleet\": \"\ue3ad\", \"LightSleetShowers\": \"\ue31a\", \"LightSnow\": \"\ue31a\", \"LightSnowShowers\": \"\ufb7d\", \"PartlyCloudy\": \"\ue30c\", \"Sunny\": \"\ue30d\", \"ThunderyHeavyRain\": \"\ufb7c\", \"ThunderyShowers\": \"\ue30e\", \"ThunderySnowShowers\": \"\ue366\", \"VeryCloudy\": \"\ue33d\" } }, \"custom/zeit\": { \"return-type\": \"json\", \"interval\": \"once\", \"format\": \"{icon}\", \"format-icons\": { \"tracking\": \"\ufab4\", \"stopped\": \"\uf236\" }, \"exec\": \"/bin/sh /usr/share/sway/scripts/zeit.sh status\", \"on-click\": \"/bin/sh /usr/share/sway/scripts/zeit.sh click; pkill -RTMIN+10 waybar\", \"exec-if\": \"[ -x \\\"$(command -v zeit)\\\" ]\", \"signal\": 10 }, \"custom/dnd\": { \"interval\": \"once\", \"return-type\": \"json\", \"format\": \"{}{icon}\", \"format-icons\": { \"default\": \"\uf868\", \"dnd\": \"\ufba1\" }, \"on-click\": \"makoctl mode | grep 'do-not-disturb' && makoctl mode -r do-not-disturb || makoctl mode -a do-not-disturb; pkill -RTMIN+11 waybar\", \"on-click-right\": \"makoctl restore\", \"exec\": \"printf '{\\\"alt\\\":\\\"%s\\\",\\\"tooltip\\\":\\\"mode: %s\\\"}' $(makoctl mode | grep -q 'do-not-disturb' && echo dnd || echo default) $(makoctl mode | tail -1)\", \"signal\": 11 }, \"custom/adaptive-light\": { \"interval\": \"once\", \"tooltip\": true, \"return-type\": \"json\", \"format\": \"{icon}\", \"format-icons\": { \"on\": \"\uf5e0\", \"off\": \"\uf5df\" }, \"exec\": \"/usr/share/sway/scripts/wluma.sh\", \"on-click\": \"/usr/share/sway/scripts/wluma.sh toggle; pkill -RTMIN+12 waybar\", \"exec-if\": \"/usr/share/sway/scripts/wluma.sh check\", \"signal\": 12 } }","title":"output max_render_time off|"},{"location":"training_pipeline/","text":"trainer.train trainer-stage train, valid, test dict_keys(['train_loss', 'valid_loss', 'valid_trues', 'valid_logits', 'valid_preds', 'valid_probs', 'valid_elapsed_time', 'val_MulticlassAccuracy', 'val_MulticlassPrecision', 'val_MulticlassRecall', 'val_MulticlassAUROC', 'val_MulticlassCalibrationError', ('val_MulticlassCalibrationError', 'train_loss')]) python tools/train.py -f exps/example/custom/nano.py -d 1 -b 8 -c yolox_nano.pth","title":"Training pipeline"},{"location":"venv/","text":"venv sudo apt install python3-venv python3 -m venv foobar source foobar/bin/activate deactivate","title":"venv"},{"location":"venv/#venv","text":"sudo apt install python3-venv python3 -m venv foobar source foobar/bin/activate deactivate","title":"venv"},{"location":"wandb/","text":"Weights & Biases Machine learning experiment tracking, dataset versioning, and model evaluation import wandb from omegaconf import DictConfig import pandas as pd class WeightsAndBiases: def __init__(self, cfg: DictConfig) -> None: self.cfg: DictConfig = cfg if cfg.debug: wandb.init(mode=\"disabled\") else: wandb.init(project=cfg.project, entity=\"peekingduck\", config=cfg) def watch(self, model) -> None: wandb.watch(model) def log(self, loss) -> None: wandb.log(loss) def log_history(self, history) -> None: selected_history = { key: history[key] for key in [ \"train_loss\", \"valid_loss\", \"valid_elapsed_time\", \"val_MulticlassAccuracy\", \"val_MulticlassPrecision\", \"val_MulticlassRecall\", \"val_MulticlassAUROC\", ] } df: pd.DataFrame = pd.DataFrame(selected_history) for row_dict in df.to_dict(orient=\"records\"): wandb.log(row_dict) def log_training_loss(self, loss) -> None: wandb.log({\"train_loss\": loss}) def log_validation_loss(self, loss) -> None: wandb.log({\"val_loss\": loss}) Your personal account has 100 GB of free storage and artifacts. Usage Pricing Storage $0.08 per GB up to 10 TB $0.06 per GB up to 100 TB $0.05 per GB up to 1000 TB Over 1000 TB, contact us Artifact tracking $0.05 per GB up to 10 TB $0.03 per GB up to 100 TB $0.02 per GB up to 1000 TB Over 1000 TB, contact us One team, up to 10 users Email and chat support 100 GB storage and artifacts tracking included. For additional storage, see prices.","title":"Model Analysis"},{"location":"wandb/#weights-biases","text":"Machine learning experiment tracking, dataset versioning, and model evaluation import wandb from omegaconf import DictConfig import pandas as pd class WeightsAndBiases: def __init__(self, cfg: DictConfig) -> None: self.cfg: DictConfig = cfg if cfg.debug: wandb.init(mode=\"disabled\") else: wandb.init(project=cfg.project, entity=\"peekingduck\", config=cfg) def watch(self, model) -> None: wandb.watch(model) def log(self, loss) -> None: wandb.log(loss) def log_history(self, history) -> None: selected_history = { key: history[key] for key in [ \"train_loss\", \"valid_loss\", \"valid_elapsed_time\", \"val_MulticlassAccuracy\", \"val_MulticlassPrecision\", \"val_MulticlassRecall\", \"val_MulticlassAUROC\", ] } df: pd.DataFrame = pd.DataFrame(selected_history) for row_dict in df.to_dict(orient=\"records\"): wandb.log(row_dict) def log_training_loss(self, loss) -> None: wandb.log({\"train_loss\": loss}) def log_validation_loss(self, loss) -> None: wandb.log({\"val_loss\": loss}) Your personal account has 100 GB of free storage and artifacts. Usage Pricing Storage $0.08 per GB up to 10 TB $0.06 per GB up to 100 TB $0.05 per GB up to 1000 TB Over 1000 TB, contact us Artifact tracking $0.05 per GB up to 10 TB $0.03 per GB up to 100 TB $0.02 per GB up to 1000 TB Over 1000 TB, contact us One team, up to 10 users Email and chat support 100 GB storage and artifacts tracking included. For additional storage, see prices.","title":"Weights & Biases"},{"location":"yolox/","text":"yolox python tools/train.py -f exps/example/custom/nano.py -d 1 -b 2 -c yolox_nano.pth 2023-04-20 08:49:15.731 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 12. 98.56714 250.26144 70.88641 9.62624] 2023-04-20 08:49:15.733 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 13. 130.93268 229.41734 91.643555 72.7481 ] 2023-04-20 08:49:16 | INFO | yolox.core.trainer:261 - epoch: 5/5, iter: 60/62, gpu mem: 0Mb, mem: 5.4Gb, iter_time: 3.179s, data_time: 2.856s, total_loss: 1.6, iou_loss: 0.6, l1_loss: 0.1, conf_loss: 0.4, cls_loss: 0.4, lr: 3.085e-04, size: 416, ETA: 0:00:06 2023-04-20 08:49:18.890 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 58. 133.07756 166.66658 5.388448 4.927013] 2023-04-20 08:49:18.890 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 6. 138.9172 213.91843 94.45138 352.15067] 2023-04-20 08:49:18.890 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 4. 218.88423 140.31888 104.04545 165.01555] 2023-04-20 08:49:18.891 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 44. 141.09167 260.7022 282.48337 301.27512] 2023-04-20 08:49:18.893 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 14. 130.99757 47.066467 56.686657 20.449068] 2023-04-20 08:49:18.895 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 64. 220.6489 303.2769 114.6341 102.452896] 2023-04-20 08:49:18.895 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 61. 256.1682 55.158962 42.048447 45.89014 ] 2023-04-20 08:49:18.895 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [24. 19.571447 57.297344 19.864103 34.632 ] 2023-04-20 08:49:18.899 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 40. 178.63289 129.9869 7.916896 7.5660305] 2023-04-20 08:49:18.901 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 66. 319.8782 183.105 111.56579 109.044 ] 2023-04-20 08:49:18.901 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 54. 269.932 236.27823 85.54 143.03235] 2023-04-20 08:49:18.901 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 49. 340.53177 187.7362 147.19244 134.72554] 2023-04-20 08:49:18.903 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 69. 283.95578 123.04812 243.99066 180.18655] 2023-04-20 08:49:18.904 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 26. 183.09532 116.72701 26.500448 15.249037] 2023-04-20 08:49:18.904 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 12. 98.56714 250.26144 70.88641 9.62624] 2023-04-20 08:49:18.905 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 13. 285.06732 229.41734 91.643555 72.7481 ] 2023-04-20 08:49:22.045 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 6. 138.0828 213.91843 94.45138 352.15067] 2023-04-20 08:49:22.045 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 58. 282.92242 166.66658 5.388448 4.927013] 2023-04-20 08:49:22.046 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 44. 242.90833 260.7022 282.48337 301.27512] 2023-04-20 08:49:22.047 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 4. 218.88423 140.31888 104.04545 165.01555] 2023-04-20 08:49:22.049 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 14. 285.00244 47.066467 56.686657 20.449068] 2023-04-20 08:49:22.050 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 64. 220.6489 303.2769 114.6341 102.452896] 2023-04-20 08:49:22.050 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 24. 292.42856 57.297344 19.864103 34.632 ] 2023-04-20 08:49:22.051 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 61. 159.83177 55.158962 42.048447 45.89014 ] 2023-04-20 08:49:22.054 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 40. 237.36711 129.9869 7.916896 7.5660305] 2023-04-20 08:49:22.056 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 49. 75.46822 187.7362 147.19244 134.72554] 2023-04-20 08:49:22.056 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 54. 269.932 236.27823 85.54 143.03235] 2023-04-20 08:49:22.058 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 69. 283.95578 123.04812 243.99066 180.18655] 2023-04-20 08:49:22.059 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 26. 183.09532 116.72701 26.500448 15.249037] 2023-04-20 08:49:22.060 | INFO | yolox.data.datasets.mosaicdetection:__getitem__:161 - [ 13. 285.06732 229.41734 91.643555 72.7481 ] 2023-04-20 08:49:22 | INFO | yolox.core.trainer:364 - Save weights to ./YOLOX_outputs/nano 0%| | 0/3 [00:00
