Skip to content

Ollama Import GGUF Models

Ollama Models#

If you're looking to experiment with various models easily, importing GGUF might be your go-to method. Here's how it works:

  • You start by creating a Modelfile, which acts as a key to unlock any GGUF model you want to use.
Modelfile
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
FROM /agentica-org_DeepCoder-14B-Preview-Q8_0.gguf
FROM /agentica-org_DeepCoder-14B-Preview.imatrix

TEMPLATE """
{{- if .System }}{{ .System }}{{ end }}
{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1}}
{{- if eq .Role "user" }}<｜User｜>{{ .Content }}
{{- else if eq .Role "assistant" }}<｜Assistant｜>{{ .Content }}{{- if not $last }}<｜end▁of▁sentence｜>{{- end }}
{{- end }}
{{- if and $last (ne .Role "assistant") }}<｜Assistant｜>{{- end }}
{{- end }}
"""
  • Once that file is ready, all you need to do is run a simple command, and voilà—your model is imported and ready for action!
ollama create deepcoder -f ./Modelfile

# gathering model components 
# copying file sha256:8add5abcbf3c0413496f039275119f6d74555a16e410c32ada75d69815d904cb 100% 
# parsing GGUF 
# using existing layer sha256:8add5abcbf3c0413496f039275119f6d74555a16e410c32ada75d69815d904cb 
# creating new layer sha256:6b62346ff5a355054f0cf583cb76c4cb0841c729114aae34d40e75c7170a8c59 
# writing manifest 
# success