Comments (13)
Awesome work @jiegec! Thanks for sharing the issues/process, this has been very useful to also understand your changes to the codebase. I also read your blog post to understand your changes (If you don't mind, I added a link to it in listed it to https://github.com/enjoy-digital/litex/wiki/Tutorials-Resources#useful-resources).
The VCU128 and Clamshell LiteDRAM/LiteX support seems fine and we'll be able to merge them. Before doing so, I just want to think a bit about the Clamshell support to LiteDRAM and see how we could eventually reduce changes in the codebase. I'll get back to you in a few days.
from litex-boards.
I isolated the module 0 via:
pads = PHYPadsReducer(platform.request("ddram"), [0]),
The calibration result:
Initializing SDRAM @0x40000000...
Switching SDRAM to software control.
Write leveling:
tCK equivalent taps: 560
Cmd/Clk scan (0-280)
|11111 |11111 |11111 |11111| best: 0
Setting Cmd/Clk delay to 0 taps.
Data scan:
m0: |111111111111111111111111| delay: 00
Write latency calibration:
m0:2
Read leveling:
m0, b00: |00000000000000000000000000000000| delays: -
m0, b01: |00000000000000000000000000000000| delays: -
m0, b02: |00000000000000000000000000000000| delays: -
m0, b03: |00000000000000000000000000000000| delays: -
m0, b04: |00000000000000000000000000000000| delays: -
m0, b05: |00000000000000000000000000000000| delays: -
m0, b06: |00000000000000000000000000000000| delays: -
m0, b07: |00000000000000000000000000000000| delays: -
best: m0, b00 delays: -
Switching SDRAM to hardware control.
Memtest at 0x40000000 (2.0MiB)...
Write: 0x40000000-0x40200000 2.0MiB
Read: 0x40000000-0x40200000 2.0MiB
bus errors: 136/256
addr errors: 0/8192
data errors: 524288/524288
Memtest KO
from litex-boards.
By reading the datasheet, module 0, 2, 4, 7 corresponds to U74 and U73. They are located in the bottom side of clam-shell.
I see: there is a dedicated CS_B pin for the bottom half in the xdc:
set_property PACKAGE_PIN BK48 [get_ports "PL_DDR4_BOT_CS_B"] ;# Bank 66 VCCO - DDR4_VDDQ_1V2 - IO_L7P_T1L_N0_QBC_AD13P_66
set_property IOSTANDARD SSTL12_DCI [get_ports "PL_DDR4_BOT_CS_B"] ;# Bank 66 VCCO - DDR4_VDDQ_1V2 - IO_L7P_T1L_N0_QBC_AD13P_66
set_property PACKAGE_PIN BP49 [get_ports "PL_DDR4_CS_B"] ;# Bank 66 VCCO - DDR4_VDDQ_1V2 - IO_L1N_T0L_N1_DBC_66
set_property IOSTANDARD SSTL12 [get_ports "PL_DDR4_CS_B"] ;# Bank 66 VCCO - DDR4_VDDQ_1V2 - IO_L1N_T0L_N1_DBC_66
However, adding BK48 to cs_n does not work:
Subsignal("cs_n", Pins("BP49 BK48"), IOStandard("SSTL12_DCI")), # Clam-shell fashion
clam-shell is not equivalent to dual-rank dimms. It is still single-rank, but has two cs pins.
from litex-boards.
I tried to replicate the cs_n signal to two pins:
# in litex-boards
Subsignal("cs_n", Pins("BP49"), IOStandard("SSTL12_DCI")),
Subsignal("bot_cs_n", Pins("BK48"), IOStandard("SSTL12_DCI")), # Clam-shell fashion
# in litedram
commands = {
# Pad name: (DFI name, Pad type (required or optional))
"reset_n" : ("reset_n", "optional"),
"cs_n" : ("cs_n", "optional"),
"bot_cs_n" : ("cs_n", "optional"),
"a" : ("address", "required"),
pads_ba : ("bank" , "required"),
"ras_n" : ("ras_n" , "required"),
"cas_n" : ("cas_n" , "required"),
"we_n" : ("we_n" , "required"),
"cke" : ("cke" , "optional"),
"odt" : ("odt" , "optional"),
"act_n" : ("act_n", "optional"),
}
But it still does not work.
from litex-boards.
After more investigation, address mirroring is applied to bottom DRAMs: https://support.xilinx.com/s/question/0D52E00006tceu6SAA/ultrascalempsoc-pl-ddr4-clamshell-address-mirroring?language=en_US
Which means we have to mirror some pins when accessing the bottom half. But does it cause the calibration to fail? UPDATE: Yes, it interfere with Mode Register Set
command.
from litex-boards.
I manually swapped the address lines:
Subsignal("a", Pins(
# "BF50 BD51 BG48 BE50 BE49 BE51 BF53 BG50",
# "BF51 BG47 BF47 BG49 BF48 BF52"),
# Swap A3A4, A5A6, A7A8, A11A13
"BF50 BD51 BG48 BE49 BE50 BF53 BE51 BF51",
"BG50 BG47 BF47 BF52 BF48 BG49"),
IOStandard("SSTL12_DCI")),
# Subsignal("ba", Pins("BE54 BE53"), IOStandard("SSTL12_DCI")),
# Swap BA0/1
Subsignal("ba", Pins("BE53 BE54"), IOStandard("SSTL12_DCI")),
Subsignal("cs_n", Pins("BK48"), IOStandard("SSTL12_DCI")), # Clam-shell fashion
#Subsignal("cs_n", Pins("BP49"), IOStandard("SSTL12_DCI")),
#Subsignal("bot_cs_n", Pins("BK48"), IOStandard("SSTL12_DCI")), # Clam-shell fashion
In this way, the bottom DRAMs can be calibrated successfully. However, the top and bottom ones can not be used at the same time.
from litex-boards.
To sum up, litedram need to be extended to support:
- Mirrored Pins: A3A4, A5A6, A7A8, A11A13, BA0BA1, BG0BG1. Only
Load Mode Register
command needs special handling, maybe only in software. - Clam shell fashion: like two ranks but with private data pins, like one rank but needs special caring.
from litex-boards.
litedram already considers address inverting for RDIMMs:
if phy_settings.is_rdimm:
assert phy_settings.memtype == "DDR4"
# JESD82-31A page 38
#
# B-side chips have certain usually-inconsequential address and BA
# bits inverted by the RCD to reduce SSO current. For mode register
# writes, however, we must compensate for this. BG[1] also directs
# writes either to the A side (BG[1]=0) or B side (BG[1]=1)
#
# The 'ba != 7' is because we don't do this to writes to the RCD
# itself.
if ba != 7:
invert_masks.append((0b10101111111000, 0b1111))
We can handle it in a similar way for address mirroring.
from litex-boards.
The DFII interface replicates the cs_n bits:
phase.cs_n.eq(Replicate(~self._command.fields.cs, len(phase.cs_n))),
phase.we_n.eq(~self._command.fields.we),
phase.cas_n.eq(~self._command.fields.cas),
phase.ras_n.eq(~self._command.fields.ras)
We need to expose the two cs_n bits to software upon calibration. In later places, the two cs_n can be used as one.
from litex-boards.
I am working on a dirty fix:
- Change csr_dfi in DFIInjector to allow software to control top/bottom cs_n separately
- In other cases, assign both cs_n to the same pin from BankMachine
- Swap address pins in software and send the same commands to top/bottom chip separately
from litex-boards.
I made it!
--========== Initialization ============--
Initializing SDRAM @0x40000000...
Switching SDRAM to software control.
Write leveling:
tCK equivalent taps: 556
Cmd/Clk scan (0-278)
|00011 |111111111 |111111111 |111111111| best: 334
Setting Cmd/Clk delay to 334 taps.
Data scan:
m0: |000111111111111111111000| delay: 37
m1: |011111111111111111100000| delay: 05
m2: |001111111111111111110000| delay: 26
m3: |001111111111111111110000| delay: 31
m4: |000011111111111111111100| delay: 64
m5: |000000011111111111111111| delay: 106
m6: |000011111111111111111000| delay: 53
m7: |000011111111111111111000| delay: 54
Write latency calibration:
m0:6 m1:6 m2:6 m3:6 m4:6 m5:6 m6:6 m7:6
Read leveling:
m0, b00: |00000000000000000000000000000000| delays: -
m0, b01: |00000000000000000000000000000000| delays: -
m0, b02: |00000000000000000000000000000000| delays: -
m0, b03: |11100000000000000000000000000000| delays: 22+-22
m0, b04: |00000000111111111111100000000000| delays: 217+-102
m0, b05: |00000000000000000000000001111111| delays: 454+-56
m0, b06: |00000000000000000000000000000000| delays: -
m0, b07: |00000000000000000000000000000000| delays: -
best: m0, b04 delays: 215+-104
m1, b00: |00000000000000000000000000000000| delays: -
m1, b01: |00000000000000000000000000000000| delays: -
m1, b02: |00000000000000000000000000000000| delays: -
m1, b03: |11111100000000000000000000000000| delays: 42+-42
m1, b04: |00000000001111111111111000000000| delays: 255+-103
m1, b05: |00000000000000000000000000001111| delays: 475+-35
m1, b06: |00000000000000000000000000000000| delays: -
m1, b07: |00000000000000000000000000000000| delays: -
best: m1, b04 delays: 255+-103
m2, b00: |00000000000000000000000000000000| delays: -
m2, b01: |00000000000000000000000000000000| delays: -
m2, b02: |00000000000000000000000000000000| delays: -
m2, b03: |11100000000000000000000000000000| delays: 15+-15
m2, b04: |00000001111111111111000000000000| delays: 203+-105
m2, b05: |00000000000000000000000001111111| delays: 447+-64
m2, b06: |00000000000000000000000000000000| delays: -
m2, b07: |00000000000000000000000000000000| delays: -
best: m2, b04 delays: 203+-107
m3, b00: |00000000000000000000000000000000| delays: -
m3, b01: |00000000000000000000000000000000| delays: -
m3, b02: |00000000000000000000000000000000| delays: -
m3, b03: |00000000000000000000000000000000| delays: -
m3, b04: |00011111111111111000000000000000| delays: 147+-108
m3, b05: |00000000000000000000011111111111| delays: 416+-95
m3, b06: |00000000000000000000000000000000| delays: -
m3, b07: |00000000000000000000000000000000| delays: -
best: m3, b04 delays: 149+-111
m4, b00: |00000000000000000000000000000000| delays: -
m4, b01: |00000000000000000000000000000000| delays: -
m4, b02: |00000000000000000000000000000000| delays: -
m4, b03: |00000000000000000000000000000000| delays: -
m4, b04: |11111111111110000000000000000000| delays: 102+-102
m4, b05: |00000000000000000111111111111100| delays: 373+-105
m4, b06: |00000000000000000000000000000000| delays: -
m4, b07: |00000000000000000000000000000000| delays: -
best: m4, b05 delays: 375+-104
m5, b00: |00000000000000000000000000000000| delays: -
m5, b01: |00000000000000000000000000000000| delays: -
m5, b02: |00000000000000000000000000000000| delays: -
m5, b03: |00000000000000000000000000000000| delays: -
m5, b04: |11111111111000000000000000000000| delays: 89+-89
m5, b05: |00000000000000001111111111111000| delays: 354+-104
m5, b06: |00000000000000000000000000000000| delays: -
m5, b07: |00000000000000000000000000000000| delays: -
best: m5, b05 delays: 354+-106
m6, b00: |00000000000000000000000000000000| delays: -
m6, b01: |00000000000000000000000000000000| delays: -
m6, b02: |00000000000000000000000000000000| delays: -
m6, b03: |00000000000000000000000000000000| delays: -
m6, b04: |00000111111111111000000000000000| delays: 169+-97
m6, b05: |00000000000000000000000111111111| delays: 432+-79
m6, b06: |00000000000000000000000000000000| delays: -
m6, b07: |00000000000000000000000000000000| delays: -
best: m6, b04 delays: 172+-100
m7, b00: |00000000000000000000000000000000| delays: -
m7, b01: |00000000000000000000000000000000| delays: -
m7, b02: |00000000000000000000000000000000| delays: -
m7, b03: |00000000000000000000000000000000| delays: -
m7, b04: |11111111111100000000000000000000| delays: 93+-93
m7, b05: |00000000000000000111111111111000| delays: 365+-102
m7, b06: |00000000000000000000000000000000| delays: -
m7, b07: |00000000000000000000000000000000| delays: -
best: m7, b05 delays: 363+-101
Switching SDRAM to hardware control.
Memtest at 0x40000000 (2.0MiB)...
Write: 0x40000000-0x40200000 2.0MiB
Read: 0x40000000-0x40200000 2.0MiB
Memtest OK
Memspeed at 0x40000000 (Sequential, 2.0MiB)...
Write speed: 108.8MiB/s
Read speed: 93.6MiB/s
--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
No boot medium found
--============= Console ================--
litex>
from litex-boards.
You can find my changes here:
https://github.com/litex-hub/litex-boards/compare/master...jiegec:litex-boards:vcu128?expand=1
https://github.com/enjoy-digital/litex/compare/master...jiegec:litex:vcu128?expand=1
https://github.com/enjoy-digital/litedram/compare/master...jiegec:litedram:vcu128?expand=1
I will upstream my changes next.
from litex-boards.
The different PRs are now merged, thanks a lot!
from litex-boards.
Related Issues (20)
- digilent_arty fails to build using f4pga due to dependency on unsupported features (DNA_PORT, XADC) HOT 4
- Sipeed Tang Nano 9k no Boot Message from SoC-Core HOT 4
- Error Related to csr.json File During Generation of Litex.resc and Litex.repl Files HOT 1
- qmtech_wukong: how do I add SPI? HOT 1
- arty 35t: SPI SDCard without PMOD HOT 1
- Sipeed Tang Nano 1k support HOT 2
- Nitefury board PCIe clock setting HOT 2
- ModuleNotFoundError: No module named 'litex' HOT 1
- efinix_titanium_ti60_f225_dev_kit: ethernet non functional
- Bitstream loading on ECPIX-5 board needs --cable ft4232 HOT 2
- Unable to replicate Analogue Pocket HOT 6
- Issues with cpu type rocket for terasic de2 115 HOT 1
- Mini901 Initial investigations - anyone interested ? HOT 1
- Sipeed Tang Mega 138k : SDRAM test failed. HOT 4
- How to make : Running Dual-Core RISC-V Linux on Cheap FPGA Board HOT 2
- limesdr_mini_v2 loading firmware: [LITEX-TERM] Got unexpected response from device 'b'\n'' HOT 7
- Sipeed Tang primer 20k : usb host test failed. HOT 5
- Sipeed Tang Primer 25k no serial out after successful flash HOT 4
- 5a-75b revision 8.2 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from litex-boards.